Welcome, Guest

Author Topic: Coding  (Read 137822 times)

vh

  • formerly mudkipz
  • *****
  • Posts: 1138
  • "giving heat meaning"
Re: Coding
« Reply #720 on: May 09, 2018, 02:39:17 PM »
so for some strange reason, no one has ever produced a robust and performant clustering algorithm for unbalanced clusters on python yet... so i had to implement my own

gt.png: the ground truth clusters. the green cluster contains 99% of all the points

scikit.png: using the standard "k means" algorithm

kc.png: using a k-centers algorithm i made up on the spot

none of scikit-learn's algorithms can really solve this task -- at least not any of the reasonably fast and well known algorithms anyway. my implementation in tensorflow runs about 20x faster than k-means by scikit-learn AND it actually solves the task.

the algorithm is pretty simple: initialize k centers using the well known 2-apx solution. then iteratively optimize the centers a la lloyd's algorithm for k means.

vh

  • formerly mudkipz
  • *****
  • Posts: 1138
  • "giving heat meaning"
Re: Coding
« Reply #721 on: July 07, 2018, 10:14:18 AM »
i have just written the most useful piece of python code ever

Code: [Select]

def iscollection(obj):
    return isinstance(obj, list) or isinstance(obj, dict) or isinstance(obj, tuple)

def apply_recursively(collection, f):

    def dict_valmap(g, dct):
        return {k:g(v) for (k,v) in dct.items()}

    def f_prime(x):
        if iscollection(x):
            return apply_recursively(x, f)
        else:
            return f(x)
   
    if not iscollection(collection):
        return f(collection)
    else:
        map_fn = dict_valmap if isinstance(collection, dict) else map
        return type(collection)(map_fn(f_prime, collection))


« Last Edit: July 07, 2018, 10:29:33 AM by vh »

vh

  • formerly mudkipz
  • *****
  • Posts: 1138
  • "giving heat meaning"
Re: Coding
« Reply #722 on: August 18, 2018, 05:10:11 PM »
i coded a ray tracer. two sample images with glass on metal

Bla

  • Global Moderator
  • *****
  • Posts: 1011
  • The stars died so you can live.
Re: Coding
« Reply #723 on: August 19, 2018, 06:06:12 AM »
Cool

FiahOwl

  • *****
  • Posts: 1234
  • This is, to give a dog and in recompense desire my dog again.
Re: Coding
« Reply #724 on: August 19, 2018, 09:49:22 AM »
Neat

tuto99

  • *****
  • Posts: 532
  • the electricity may be cut by something stimulus.
Re: Coding
« Reply #725 on: August 19, 2018, 11:38:43 AM »
Nice

vh

  • formerly mudkipz
  • *****
  • Posts: 1138
  • "giving heat meaning"
Re: Coding
« Reply #726 on: August 19, 2018, 07:30:09 PM »
Good

atomic7732

  • Global Moderator
  • *****
  • Posts: 3770
  • caught in the river turning blue
    • Paladin of Storms
Re: Coding
« Reply #727 on: August 20, 2018, 11:16:49 AM »
Like

Darvince

  • *****
  • Posts: 1814
  • gab
Re: Coding
« Reply #728 on: August 20, 2018, 12:54:12 PM »
Thumbsup

tuto99

  • *****
  • Posts: 532
  • the electricity may be cut by something stimulus.
Re: Coding
« Reply #729 on: August 20, 2018, 04:58:14 PM »
Subscribed

vh

  • formerly mudkipz
  • *****
  • Posts: 1138
  • "giving heat meaning"
Re: Coding
« Reply #730 on: December 07, 2018, 12:39:40 PM »
my emails-sent-per-month plot has gotten more advanced

new features include formatted dates on the x axis, which was surprisingly difficult
and also an end-of-month projection for the current month, in red


vh

  • formerly mudkipz
  • *****
  • Posts: 1138
  • "giving heat meaning"
Re: Coding
« Reply #731 on: December 07, 2018, 12:56:14 PM »
i can't tell if the colors make this more insightful or just more confusing

green = summer break
turqoise = fall semester
purple = spring semester


vh

  • formerly mudkipz
  • *****
  • Posts: 1138
  • "giving heat meaning"
Re: Coding
« Reply #732 on: December 29, 2018, 10:31:52 AM »
a minimal cuda project -- it's always annoying to have to remember how to set these up, which is why i've done so right here.


vh

  • formerly mudkipz
  • *****
  • Posts: 1138
  • "giving heat meaning"
Re: Coding
« Reply #733 on: February 02, 2019, 12:29:49 AM »
i labeled my axes and have a legend so you should be able to figure out what this plot is of

for konisshi <-> spawntown, which lies at roughly (1000, 100) on this graph, it's close to a tie between overworld elytra, nether elytra, and nether ice roads. however since spawntown nether hub is quite clunky, i think overworld elytra ends up being significantly faster

for konisshi <-> laplaya, which lies at maybe (4000, 1000) on this graph, nether elytra clearly wins.

for konisshi <-> yqtland, which i have no idea how far it is let's say it's at (300, 1000) on this graph, overworld elytra wins

konisshi <-> darvincia happens like 5 times a year, so let's say (3000, 10), so nether elytra and overworld elytra are pretty close

if for some reason i went from world-corner to world-corner twice a day for a year, that would be about (22000, 1000), that would be a legit reason to build a nether ice boat route

and finally to get from one end of my storage to the other end, 40 blocks away, and something i do constantly, beacon sprinting is the fastest, as expected

factors this graph takes into account
1. how long it takes to build the infrastructure for a given mode of transport, which consists of a one-time-cost plus a cost-per-meter
2. how long it takes to use infrastructure, which includes a per-trip-cost (such as the 10 seconds you pay to go through two portals) as well as a per-distance cost
3. the "efficiency" of a method -- ice boat has less than 1 efficiency since it's hard to make turns. elytra in nether also has < 1 efficiency due to the fact that paths aren't perfectly straight.
« Last Edit: February 02, 2019, 12:47:13 AM by vh »

Bla

  • Global Moderator
  • *****
  • Posts: 1011
  • The stars died so you can live.
Re: Coding
« Reply #734 on: February 02, 2019, 05:16:13 AM »
Interesting, though it's missing all our glorious rail lines

vh

  • formerly mudkipz
  • *****
  • Posts: 1138
  • "giving heat meaning"
Re: Coding
« Reply #735 on: February 02, 2019, 08:53:37 AM »
they didn't show up because they are strictly worse than elytra. maybe when mojang puts high speed rail back in the game

vh

  • formerly mudkipz
  • *****
  • Posts: 1138
  • "giving heat meaning"
Re: Coding
« Reply #736 on: February 26, 2019, 06:53:59 PM »
ubv now supports SASL, a method of authentication which allows ubv to be hosted at ip addresses which are othrewise blacklisted by freenode

here's the relevant bits of code, which are definitely not guaranteed to be correct but it works for me

Code: [Select]
    def initialize(self):
       
        self.send("CAP REQ :sasl")       
        self.send('NICK %s' % conf.NICK)
        self.send('USER %s %s %s: Hello World' % (conf.NICK, conf.NICK, conf.NICK))

        while u"ACK :sasl" not in self.receive():
            pass

        self.send("AUTHENTICATE PLAIN")

        while u"AUTHENTICATE +" not in self.receive():
            pass

        key = '%s\0%s\0%s' % (conf.NICK, conf.NICK, conf.SASL_PW)
        auth_msg = bytes('AUTHENTICATE '+base64.b64encode(key.encode('utf8').decode('utf8'))+'\n')
        print 'sending', auth_msg
        self.s.send(auth_msg)

        while u"authentication successful" not in self.receive():
            pass

        self.send("CAP END")
        self.send('JOIN %s' % conf.CHANNEL)

the reason this is important is that i'll be moving around for the next few months and my desktop is unlikely to be reliably on, so ubv is now hosted on another aws instance but wait you say doesn't it kind of defeat the point of having blacraft on a temporarily hosted aws instance if you're putting ubv on a permanently hosted instance? well no, because ubv runs on the lightest instance possible, with 487M of ram and something like 1/10th of a cpu core

atomic7732

  • Global Moderator
  • *****
  • Posts: 3770
  • caught in the river turning blue
    • Paladin of Storms
Re: Coding
« Reply #737 on: March 08, 2019, 01:02:01 AM »
coding a sea surface temperature simulator so that i can have some inputs for wrf that make sense on pavala, as sea surface temperatures play a huge part of atmospheric forcing and simulating pavala with pavalan land and earth sea surface temperatures provides a worthless result, basically

it doesn't really simulate radiation or advection very well so it's just a shallow water simulation right now, that means there isn't like a sea surface temperature variable being advected around, hence the uniformly colored light blue background

i'm really amazed that with a simple initialization of a symmetric low height ("pressure", kinda) region about the equator you get slow westward propagation of a rossby wave and fast eastward propagation of a kelvin wave

we've been learning about equatorial waves in my tropical meteorology class so it's really cool to see that with basically 3 equations i can see them in action... and like... i made them!

currently wrapping only works going from the eastern hemisphere to the west (so that the kelvin wave can go around), so when the rossby wave reaches the western boundary toward the end of the run it creates weird artifacts

you also get to see the model break down and do weird stuff and i'm not entirely sure what that's all about, but the main important thing to see is that the kelvin wave makes it around the planet almost 1.5 times in the same time (20 days) the rossby wave only goes about halfway, which is about what you would expect.



contours are height/layer depth, arrows are flow in the layer
« Last Edit: March 08, 2019, 01:15:13 AM by atomic7732 »

vh

  • formerly mudkipz
  • *****
  • Posts: 1138
  • "giving heat meaning"
Re: Coding
« Reply #738 on: March 13, 2019, 06:12:59 PM »

vh

  • formerly mudkipz
  • *****
  • Posts: 1138
  • "giving heat meaning"
Re: Coding
« Reply #739 on: April 08, 2019, 01:14:50 AM »
coding challenge: write an algorithm that determines if a sigmar's garden game (from opus magnum) is solvable, and if so, then solve it

vh

  • formerly mudkipz
  • *****
  • Posts: 1138
  • "giving heat meaning"
Re: Coding
« Reply #740 on: April 16, 2019, 05:45:13 PM »
so i've been doing some lagbusting on blacraft

first i measured tps while loading various sets of chunks in blacraft, making 36 measurements, each covering a different subset of 28 predefined "subregions".



then the mspt (milliseconds per tick) is described by mspt = Ax + noise, and the cost is |mspt - Ax|
i used l1 norm (laplacian noise model) because the measurement error is probably pretty heavy tailed
A is the matrix of coefficients which contains information on which subregions were loaded for each measurement

there are some more details and details, such as the fact that mspt under 50 is not measurable, mspt is always positive, etc, which i took into account

after optimizing the cost, this is what i get

each row is a separate measurement
column 1: the cost associated with this measurement
column 2: the expected mspt for this measurement
column 3: the actual mspt for this measurement
column 4: the error in mspt
column 5/6: the expected and actual tps

anyway if you compare the predicted and actual results they're actually pretty close -- under 0.5 tps error in almost all cases, which i'm sure is less than my measurement error

Code: [Select]
[[  0.49   50.     50.      0.     20.     20.   ]
 [ 12.603 136.241 125.     11.241   7.34    8.   ]
 [  1.64   58.201  57.143   1.058  17.182  17.5  ]
 [  0.691  62.434  62.5    -0.066  16.017  16.   ]
 [  1.328  70.809  71.429  -0.62   14.123  14.   ]
 [ 10.701  73.366  83.333  -9.967  13.63   12.   ]
 [  1.191  70.947  71.429  -0.481  14.095  14.   ]
 [  0.564  51.333  51.282   0.051  19.481  19.5  ]
 [  0.465  50.     50.      0.     20.     20.   ]
 [  2.578  50.     52.083  -2.083  20.     19.2  ]
 [  2.741  72.054  74.074  -2.02   13.879  13.5  ]
 [  2.867  60.235  62.5    -2.265  16.602  16.   ]
 [  9.116  68.492  76.923  -8.431  14.6    13.   ]
 [  0.804  53.788  54.054  -0.267  18.592  18.5  ]
 [  0.431  50.     50.      0.     20.     20.   ]
 [  1.606  52.89   51.813   1.077  18.907  19.3  ]
 [  1.216  51.935  52.632  -0.696  19.255  19.   ]
 [  7.083  80.354  74.074   6.279  12.445  13.5  ]
 [  0.074  50.     50.      0.     20.     20.   ]
 [  1.606  94.578  95.238  -0.66   10.573  10.5  ]
 [  0.165  50.     50.      0.     20.     20.   ]
 [  0.208  50.     50.      0.     20.     20.   ]
 [  0.543  50.     50.251  -0.251  20.     19.9  ]
 [  2.949  50.     52.632  -2.632  20.     19.   ]
 [  1.575  50.     51.282  -1.282  20.     19.5  ]
 [  0.097  50.     50.      0.     20.     20.   ]
 [  0.049  50.     50.      0.     20.     20.   ]
 [  0.078  50.     50.      0.     20.     20.   ]
 [  1.586  50.     51.282  -1.282  20.     19.5  ]
 [  0.186  50.     50.      0.     20.     20.   ]
 [  0.52   50.     50.251  -0.251  20.     19.9  ]
 [  0.121  50.     50.      0.     20.     20.   ]
 [  0.014  50.     50.      0.     20.     20.   ]
 [  0.112  50.     50.      0.     20.     20.   ]
 [  0.103  50.     50.      0.     20.     20.   ]
 [  0.638  50.     50.251  -0.251  20.     19.9  ]]

this also gives the mspt cost of each subregion -- remember each measurement consists of a set of subregions, so this is a decomposition of where that cost comes from
Code: [Select]
[ 1.193  9.595 33.261  6.168  5.874  5.641  0.187 -0.     0.374  3.268
 13.511  0.61   0.007  0.012  0.634  4.804  4.96   1.652  2.778  0.001
 15.503  3.059  3.746  0.574  0.703  6.552 12.687  6.248]

the first 4 subregions are in order
1. 441 chunks in the end void
2. spawntown spawn chunks
3. 21x21 chunk region in spawntown minus spawn chunks (a donut shape)
4. 500ish chunks of random terrain

and the rest of the subregions are actually a grid pattern over konisshi, which was the main target of this exercise. each of the rest of the subregions are pretty much 5x5 chunks of konisshi.

you can see that all of konisshi adds up to almost 94 mspt, which is pretty hefty. spawntown by comparison is a lightweight 43 mspt (although i'm comparing 540 chunks of konisshi to 289 of spawntown here)

i've attached a plot of these subregions in the correct geographic orientation, and you can see most of the lag is concentrated in a few of these subregions (yellower is laggier)

also attached code, which contains the Greatest List Comphrenension of All Time

Code: [Select]
    measurement_regions = [
        [item for sublist in [subregion_map[x] for x in region] for item in sublist]
        for region in measurement_regions
    ]
« Last Edit: April 16, 2019, 06:20:56 PM by vh »