MooMooMilkParty

Edit: Filtering out sodium and potassium to try to account for some of the low-wavelength counts doesn't seem to help fitting the distributions either: https://imgur.com/dHDUS9Z

You can get the data here: https://physics.nist.gov/PhysRefData/ASD/lines_form.html

And here's the (garbage) code to reproduce my plots:

    %pylab inline
    import pandas as pd
    import scipy
    import scipy.stats
    
    mpl.style.use('seaborn-muted')
    mpl.rcParams['figure.figsize'] = (18, 12)
    mpl.rcParams['font.size'] = 16
    df = pd.read_csv('nist.csv')
    
    def filter_string(x):
        x = str(x).replace('=', '').replace('"', '').replace('*', '').replace('+', '').replace('(', '').replace(')', '')
        if len(x) == 0:
            x = 'NaN'
        return x
    
    asl_comp = df['ritz_wl_vac(nm)'].apply(filter_string).astype(float)
    asl_ver  = df['obs_wl_vac(nm)'].apply(filter_string).astype(float)
    sp_num = df['sp_num'].astype(float)
    
    df_neutral = df[sp_num == 1]
    asl_comp = df_neutral['ritz_wl_vac(nm)'].apply(filter_string).astype(float)
    asl_ver  = df_neutral['obs_wl_vac(nm)'].apply(filter_string).astype(float)
    sp_num = df_neutral['sp_num'].astype(float)
    
    asl_ver = asl_ver[~asl_ver.isna()]
    
    def planck(wav, T=9000.0):
        h = 6.626e-34
        c = 3.0e+8
        k = 1.38e-23
        a = 2.0*h*c**2
        b = h*c/(wav*k*T)
        intensity = (a / ( (wav**5)) * (1 / (np.exp(b) - 1.0) ))
        return intensity
    
    
    dist_names = ['beta', 'lognorm', 'pearson3', 'gumbel_r']
    x = np.arange(0, 1500, 5)
    intensity = planck(x / 1e9)
    size = len(asl_ver)
    
    yvals, xvals = np.histogram(asl_ver, bins=300, normed=True)
    
    for dist_name in dist_names:
        dist = getattr(scipy.stats, dist_name)
        param = dist.fit(asl_ver)
        pdf_fitted = dist.pdf(x, *param[:-2], loc=param[-2], scale=param[-1])
        plt.plot(x, pdf_fitted, label=dist_name, linewidth=4,)
    
    plt.hist(asl_ver, bins=300, density=True, color='grey');
    plt.plot(x, intensity / 1.1e17, linewidth=4, color='red', linestyle='--', label='scaled planck (9000k)')
    plt.legend()
    plt.xlabel(r'$\lambda (nm)$')
    plt.ylabel('Density')

Comment · 7 years ago

Aha! As far as persisting calculations, this makes sense. I guess in my head I was thinking "pure" data where everything opened was precomputed. Forgive my ignorance about database solutions, but do they not implement something similar?

But now you've got me thinking, it would be nice if libraries like Dask could allow for flagging of symbolic operations like this to be written to disk for quickly saving metadata where intermediate steps don't explicitly need to be saved.

Comment · 7 years ago

You're blowing my mind here a little bit. I don't use MySQL but do use Excel and the R/Python data science stacks quite a bit. I regularly open data with tens or even hundreds of thousands of columns in the latter by using lazy computations. Is this not the case with database approaches? What kind of hardware are we talking? Presumably Excel has to load everything into memory at once since it's immediately viewable, right?

Comment · 8 years ago

I hope this capability gets to maturity. It consistently seems like anything I want in a language is in progress in Julia; new scientists in 15 years are going to be very spoiled if Julia becomes mainstream.