Coding

[vh]

so for some strange reason, no one has ever produced a robust and performant clustering algorithm for unbalanced clusters on python yet... so i had to implement my own

gt.png: the ground truth clusters. the green cluster contains 99% of all the points

scikit.png: using the standard "k means" algorithm

kc.png: using a k-centers algorithm i made up on the spot

none of scikit-learn's algorithms can really solve this task -- at least not any of the reasonably fast and well known algorithms anyway. my implementation in tensorflow runs about 20x faster than k-means by scikit-learn AND it actually solves the task.

the algorithm is pretty simple: initialize k centers using the well known 2-apx solution. then iteratively optimize the centers a la lloyd's algorithm for k means.

[vh]

i have just written the most useful piece of python code ever

Code: [Select]


def iscollection(obj):
    return isinstance(obj, list) or isinstance(obj, dict) or isinstance(obj, tuple)

def apply_recursively(collection, f):

    def dict_valmap(g, dct):
        return {k:g(v) for (k,v) in dct.items()}

    def f_prime(x):
        if iscollection(x):
            return apply_recursively(x, f)
        else:
            return f(x)
   
    if not iscollection(collection):
        return f(collection)
    else:
        map_fn = dict_valmap if isinstance(collection, dict) else map
        return type(collection)(map_fn(f_prime, collection))




[vh]

i coded a ray tracer. two sample images with glass on metal

[Bla]

Cool

[FiahOwl]

This message is only viewable with Universe Sandbox Galaxy Edition. Access it and much more with promo-code '195326'.

[tuto99]

Nice

[vh]

Good

[atomic7732]

Like

[Darvince]

Thumbsup

[tuto99]

Subscribed

[vh]

my emails-sent-per-month plot has gotten more advanced

new features include formatted dates on the x axis, which was surprisingly difficult
and also an end-of-month projection for the current month, in red

[vh]

i can't tell if the colors make this more insightful or just more confusing

green = summer break
turqoise = fall semester
purple = spring semester

[vh]

a minimal cuda project -- it's always annoying to have to remember how to set these up, which is why i've done so right here.

[vh]

i labeled my axes and have a legend so you should be able to figure out what this plot is of

for konisshi <-> spawntown, which lies at roughly (1000, 100) on this graph, it's close to a tie between overworld elytra, nether elytra, and nether ice roads. however since spawntown nether hub is quite clunky, i think overworld elytra ends up being significantly faster

for konisshi <-> laplaya, which lies at maybe (4000, 1000) on this graph, nether elytra clearly wins.

for konisshi <-> yqtland, which i have no idea how far it is let's say it's at (300, 1000) on this graph, overworld elytra wins

konisshi <-> darvincia happens like 5 times a year, so let's say (3000, 10), so nether elytra and overworld elytra are pretty close

if for some reason i went from world-corner to world-corner twice a day for a year, that would be about (22000, 1000), that would be a legit reason to build a nether ice boat route

and finally to get from one end of my storage to the other end, 40 blocks away, and something i do constantly, beacon sprinting is the fastest, as expected

factors this graph takes into account
1. how long it takes to build the infrastructure for a given mode of transport, which consists of a one-time-cost plus a cost-per-meter
2. how long it takes to use infrastructure, which includes a per-trip-cost (such as the 10 seconds you pay to go through two portals) as well as a per-distance cost
3. the "efficiency" of a method -- ice boat has less than 1 efficiency since it's hard to make turns. elytra in nether also has < 1 efficiency due to the fact that paths aren't perfectly straight.

[Bla]

Interesting, though it's missing all our glorious rail lines

[vh]

they didn't show up because they are strictly worse than elytra. maybe when mojang puts high speed rail back in the game

[vh]

ubv now supports SASL, a method of authentication which allows ubv to be hosted at ip addresses which are othrewise blacklisted by freenode

here's the relevant bits of code, which are definitely not guaranteed to be correct but it works for me

Code: [Select]

    def initialize(self):
       
        self.send("CAP REQ :sasl")       
        self.send('NICK %s' % conf.NICK)
        self.send('USER %s %s %s: Hello World' % (conf.NICK, conf.NICK, conf.NICK))

        while u"ACK :sasl" not in self.receive():
            pass

        self.send("AUTHENTICATE PLAIN")

        while u"AUTHENTICATE +" not in self.receive():
            pass

        key = '%s\0%s\0%s' % (conf.NICK, conf.NICK, conf.SASL_PW)
        auth_msg = bytes('AUTHENTICATE '+base64.b64encode(key.encode('utf8').decode('utf8'))+'\n')
        print 'sending', auth_msg
        self.s.send(auth_msg)

        while u"authentication successful" not in self.receive():
            pass

        self.send("CAP END")
        self.send('JOIN %s' % conf.CHANNEL)

the reason this is important is that i'll be moving around for the next few months and my desktop is unlikely to be reliably on, so ubv is now hosted on another aws instance but wait you say doesn't it kind of defeat the point of having blacraft on a temporarily hosted aws instance if you're putting ubv on a permanently hosted instance? well no, because ubv runs on the lightest instance possible, with 487M of ram and something like 1/10th of a cpu core

[atomic7732]

coding a sea surface temperature simulator so that i can have some inputs for wrf that make sense on pavala, as sea surface temperatures play a huge part of atmospheric forcing and simulating pavala with pavalan land and earth sea surface temperatures provides a worthless result, basically

it doesn't really simulate radiation or advection very well so it's just a shallow water simulation right now, that means there isn't like a sea surface temperature variable being advected around, hence the uniformly colored light blue background

i'm really amazed that with a simple initialization of a symmetric low height ("pressure", kinda) region about the equator you get slow westward propagation of a rossby wave and fast eastward propagation of a kelvin wave

we've been learning about equatorial waves in my tropical meteorology class so it's really cool to see that with basically 3 equations i can see them in action... and like... i made them!

currently wrapping only works going from the eastern hemisphere to the west (so that the kelvin wave can go around), so when the rossby wave reaches the western boundary toward the end of the run it creates weird artifacts

you also get to see the model break down and do weird stuff and i'm not entirely sure what that's all about, but the main important thing to see is that the kelvin wave makes it around the planet almost 1.5 times in the same time (20 days) the rossby wave only goes about halfway, which is about what you would expect.

https://www.youtube.com/watch?v=h-lv_D14T14

contours are height/layer depth, arrows are flow in the layer

[vh]

seems like this would make for a fun computation/graphics project

https://en.wikipedia.org/wiki/Kuramoto_model#/media/File:Spatial_synchronization_patterns_in_variants_of_the_Kuramoto_Model.png

[vh]

coding challenge: write an algorithm that determines if a sigmar's garden game (from opus magnum) is solvable, and if so, then solve it

[vh]

so i've been doing some lagbusting on blacraft

first i measured tps while loading various sets of chunks in blacraft, making 36 measurements, each covering a different subset of 28 predefined "subregions".



then the mspt (milliseconds per tick) is described by mspt = Ax + noise, and the cost is |mspt - Ax|
i used l1 norm (laplacian noise model) because the measurement error is probably pretty heavy tailed
A is the matrix of coefficients which contains information on which subregions were loaded for each measurement

there are some more details and details, such as the fact that mspt under 50 is not measurable, mspt is always positive, etc, which i took into account

after optimizing the cost, this is what i get

each row is a separate measurement
column 1: the cost associated with this measurement
column 2: the expected mspt for this measurement
column 3: the actual mspt for this measurement
column 4: the error in mspt
column 5/6: the expected and actual tps

anyway if you compare the predicted and actual results they're actually pretty close -- under 0.5 tps error in almost all cases, which i'm sure is less than my measurement error

Code: [Select]

[[  0.49   50.     50.      0.     20.     20.   ]
 [ 12.603 136.241 125.     11.241   7.34    8.   ]
 [  1.64   58.201  57.143   1.058  17.182  17.5  ]
 [  0.691  62.434  62.5    -0.066  16.017  16.   ]
 [  1.328  70.809  71.429  -0.62   14.123  14.   ]
 [ 10.701  73.366  83.333  -9.967  13.63   12.   ]
 [  1.191  70.947  71.429  -0.481  14.095  14.   ]
 [  0.564  51.333  51.282   0.051  19.481  19.5  ]
 [  0.465  50.     50.      0.     20.     20.   ]
 [  2.578  50.     52.083  -2.083  20.     19.2  ]
 [  2.741  72.054  74.074  -2.02   13.879  13.5  ]
 [  2.867  60.235  62.5    -2.265  16.602  16.   ]
 [  9.116  68.492  76.923  -8.431  14.6    13.   ]
 [  0.804  53.788  54.054  -0.267  18.592  18.5  ]
 [  0.431  50.     50.      0.     20.     20.   ]
 [  1.606  52.89   51.813   1.077  18.907  19.3  ]
 [  1.216  51.935  52.632  -0.696  19.255  19.   ]
 [  7.083  80.354  74.074   6.279  12.445  13.5  ]
 [  0.074  50.     50.      0.     20.     20.   ]
 [  1.606  94.578  95.238  -0.66   10.573  10.5  ]
 [  0.165  50.     50.      0.     20.     20.   ]
 [  0.208  50.     50.      0.     20.     20.   ]
 [  0.543  50.     50.251  -0.251  20.     19.9  ]
 [  2.949  50.     52.632  -2.632  20.     19.   ]
 [  1.575  50.     51.282  -1.282  20.     19.5  ]
 [  0.097  50.     50.      0.     20.     20.   ]
 [  0.049  50.     50.      0.     20.     20.   ]
 [  0.078  50.     50.      0.     20.     20.   ]
 [  1.586  50.     51.282  -1.282  20.     19.5  ]
 [  0.186  50.     50.      0.     20.     20.   ]
 [  0.52   50.     50.251  -0.251  20.     19.9  ]
 [  0.121  50.     50.      0.     20.     20.   ]
 [  0.014  50.     50.      0.     20.     20.   ]
 [  0.112  50.     50.      0.     20.     20.   ]
 [  0.103  50.     50.      0.     20.     20.   ]
 [  0.638  50.     50.251  -0.251  20.     19.9  ]]

this also gives the mspt cost of each subregion -- remember each measurement consists of a set of subregions, so this is a decomposition of where that cost comes from

Code: [Select]

[ 1.193  9.595 33.261  6.168  5.874  5.641  0.187 -0.     0.374  3.268
 13.511  0.61   0.007  0.012  0.634  4.804  4.96   1.652  2.778  0.001
 15.503  3.059  3.746  0.574  0.703  6.552 12.687  6.248]

the first 4 subregions are in order
1. 441 chunks in the end void
2. spawntown spawn chunks
3. 21x21 chunk region in spawntown minus spawn chunks (a donut shape)
4. 500ish chunks of random terrain

and the rest of the subregions are actually a grid pattern over konisshi, which was the main target of this exercise. each of the rest of the subregions are pretty much 5x5 chunks of konisshi.

you can see that all of konisshi adds up to almost 94 mspt, which is pretty hefty. spawntown by comparison is a lightweight 43 mspt (although i'm comparing 540 chunks of konisshi to 289 of spawntown here)

i've attached a plot of these subregions in the correct geographic orientation, and you can see most of the lag is concentrated in a few of these subregions (yellower is laggier)

also attached code, which contains the Greatest List Comphrenension of All Time

Code: [Select]

    measurement_regions = [
        [item for sublist in [subregion_map[x] for x in region] for item in sublist]
        for region in measurement_regions
    ]


[vh]

i am coding optics sim



how does it work? basically it's a limited form of raytracing in 2D, where i use newton raphson to compute the intersection between arbitrary differentiable surfaces (lens surfaces) and rays. i know the ordering of the lens components, so i can just process them sequentially.

here i show light passing through an aspherical lens with a refractive index of 2 (really the "lens" is just composed of two separate surfaces, the first with a refraction *ratio* of 2 and the second 0.5.) and then hitting a plane.



code is complete hacky garbage btw

[vh]

today's OPTIK sim upgrades include

1. refactoring the code completely so it's guud and ez to work with instead of hacky
2. adding in an aperture object which blocks out of bound light rays
3. refining the plotting

[tuto99]

thats pretty cool

[Darvince]

this is an excellent dialogue between man and nature. thank you for providing this informatical stimulus to us.

[vh]

thats pretty cool

thank

this is an excellent dialogue between man and nature. thank you for providing this informatical stimulus to us.

Oh! Oh! I know why!





3 updates today

1. code for automatically generating light sources which point in the right direction (aka into the lenses) so that it's less painful to set up simulations

2. some trickery with adaptive newton raphson step sizes, so for this simulation with 9 surfaces and roughly 1k rays the time went from 250ms down to 18ms, and with increased precision too

3. most importantly, rays have wavelength, plotting converts wavelength to rgb!!!, AND sellmeier dispersion is implemented, resulting in physically accurate chromatic aberration

[atomic7732]

simulate light rays reflecting inside a raindrop to create a rainbow

[vh]

TODAY on optik sim we have a few more upgrades

1. native support for geometric planes. previously i'd only implemented a class of spherical surfaces with quadratic and quartic asphericities. this very general class of surfaces includes a simple plane of course, but also is computationally much more expensive.

2. with the help of blazing fast planar geometry from [1], i implemented autofocus -- a search algorithm used to move the camera sensor back and forth to minimize the variance of where rays hit on the sensor. the main importance of this is that i can start designing optical systems without having to painfully manually focus the setup

3. my previous experiments with chromatic aberration used a made up material with ridiculous dispersion parameters, because chromatic aberrations are actually generally quite small and hard to see. but now, i've reverted back to using typical borosilicate glass, and i've upgraded the plotting code to show super zoomed in views of exactly where each light source hits the sensor (so even small aberrations are visible)!

[vh]

TODAY ok optik sim we have a huge image -- seriously, this thing is almost 5k by 5k, and i rendered this at half of full res just to make uploading less painful.

1. i set up a standard set of test cases for testing lenses: there are objects at 3 different distances, and 3 different angles relative to the camera, making for 9 objects. i focus the lens to the same 3 different distances, which gives me 27 triplets of (angle, distance, focus point).

2. i wrote code to plot all of this nonsense. i also added code to plot a coarser version of the light (visible in the big 3 subplots on the left), since i think it's useful to be able to see exactly how the rays travel through the system.

3. i fixed some numerical problems with the ray tracing math

interpreting the plot:
first, mentally break down the plot into 3 chunks on the vertical axis, with each chunk containing a big plot of the entire system and 9 small plots on the right.

the 3 chunks correspond to the 3 focus distances: in the first vertical chunk, the camera is focused to the closest object (0.5 m), in the second vertical chunk, 5m, and the third, 100m.

for each focus distance, i simulate how light from all 9 objects travel through the lens. however, in order to keep the figure readable, in the left subplots, i visualize only the 3 lights which are located at the same distance which the lens is focused on.

ON the right 9 subplots, i do visualize how all 9 objects look like as they hit the sensor. each COLUMN of the 9 subplots corresponds to a different distance: the 3 objects at 0.5m, the 3 at 5m, the 3 at 100m. each ROW of the 9 subplots corresponds to a different angle.

so to make this clear: in the first chunk, i would expect the middle row of the first col to be the sharpest, because that's what i'm focusing on. i would still hope that the other rows of the first col are pretty sharp. finally, i don't expect the second and third cols (objects at 5 and 100m away) to be sharp. likewise, in the second chunk, i would expect the middle row of the Second col to be sharp, etc.

each of the 9 small subplots covers an area of 1mm x 1mm. on a typical camera, there are probably somewhere in the ballpark of 50 pixels / mm. so you can see this made up lens is pretty shit because even the best focused points of light (i call them objects but they're really point light sources), are spread out over like 10px

anyway i feel like almost all of the setup is now complete and i can start designing lenses soon

-----

ALSO if you're wondering why the right plots have color but the left rays are white, it's because on the left, you're zoomed out enough that the R, G, and B rays blend together into white!, however if you zoom in enough, even on the left you can see slight chromatic aberrations leading to colors

[vh]

today we have some Real World Optics.





i generated a picture of a checkerboard, took a bunch of pictures with my phone. then i used it to jointly optimize homographies and distortion parameters of my phone's camera lens



this is a picture of some perfectly horizontal stripes i found on google images
left: a photo i took of my monitor of those stripes
right: same photo, but corrected for lens distortion