Stew Polley Coding

Scripts to help setup your server

2018-02-24T21:17:48+00:00

Setting up your server to be correctly configured to accept traffic and proxy it back to the correct applications can be difficult at times.

Ok it’s not really that difficult, especially with Digital ocean providing lovely guides for free on basicaly… anything!

But these can take time to get setup, which is why a number of automated tools exist. Mainly for a learning excercise for myself, I wanted to build my own automation to get the type of servers I was commonly using quickly set up, to help improve my own productivity.

So far I’ve written two main scripts, but will expand on these in time:

LEMP stack setup
WordPress install

You can view the detailed instructions for these scripts at https://scripts.stewpolley.com/ and view the source code on github.

What I was reading – Jan 2018 Edition

2018-01-31T20:00:16+00:00

Welcome to the first monthly edition of my “What I’m reading” series. Click here to read other posts in this series.

I’ve been compiling this list throughout January, and the following links appear in the chronological order of when I read them. Unlike my “Year in review” post, I’ll be posting links to what I read this month, not to things that were published this month.

Without further ado, here’s my reading list for Jan 2018.

Bug Bounty Guide: Tips for Bug Hunters and Product Owners alike on how to hunt for bugs, and how to operate a bug bounty system.
Everything you need to know about HTTP security headers: A run down on the different HTTP security headers you can use, what they do, and instructions on how to set them in a number of frameworks/servers
Accessibility Checklist: A look at making sure your web-pages are designed in a way to allow those with accessibility needs can easily navigate your site.
Android UI Automation: This series by the Slack Engineering team is an interesting look into how they get their UI testing done, and why they feel it’s valuable.
Outsmarting Icognito Detection: Sites are trying to force you to not use incognito, so they can track you. This post looks at one way of how to get around that.
xkcd: Meltdown and Spectre: One of the shortest and easiest to understand summaries of the whole Meltdown/Spectre incident.
Setting up a DNS Firewall on Steroids: Something I want to do in the future.

Hopefully you find one, or all of these a good, worthwhile read.

Django: Setting Cookie and returning a rendered template – an example

2018-01-13T08:27:00+00:00

While trying to figure out how to set cookies with Django (something I’ve been lucky enough to not have to do) I was of course searching “How to set a cookie with Django” this was the first result I found. Now that’s a great link and all, and it points to the relevant Django Docs.

However, there was a particular example missing. Returning a rendered template, with a cookie. Fortunately, it didn’t take long to figure it out, but I felt the fact that an exact example of this was missing as a little annoying. So for anyone else wondering “How do I return a rendered template AND set a cookie” here’s an example.

from django.shortcuts import render

def my_view(request):
    response = render(request, 'index.html')
    response.set_cookie('my_cookie', 'Here is my cookie!')
    return response

With any luck this will help at least one other person out there!

Of course, please read up the set_cookie and set_signed_cookie docs for more info on what you can do here.

Introducing StewPolley Tools

2018-01-06T06:23:07+00:00

As mentioned in my “year in review” post, I mentioned wanting to release tools that others may find useful in their day to day lives.

I am glad to say that today marks the start of that, with https://tools.stewpolley.com/ going live today. My first tool I’m calling “Timezone Aligner” – It’s the same as many other tools out there, I can hardly take credit for an original design. The other tools out there didn’t suit my needs though. Living in Queensland (AEST), we don’t have Daylight savings. Many of the other Timezone converters out there don’t take this into account, and when you try to convert Australian Eastern Standard Time to Central Standard Time, if the other Australian states are observing Daylight savings, you simply couldn’t get these other tools to work.

This tool takes this into account, and I like to think it’s a nice clean interface. I’m not a front end designer though, so feel free to tell me I’m wrong about that!

You can access my Timezone Aligner here.

Allowing HTML Purifier to accept the details and summary tags.

2018-01-05T07:48:09+00:00

The HTML Purifier library is a great way to filter user-supplied HTML, normally from WYSIWYG Rich Text editors, and remove potentially malicious content. It can also be used to ensure that certain elements always get saved with particular attribues. Ie, ensuring any user-supplied link has target="_blank" and rel="nofollow". Depending on your use case, you may even want to prevent users from inserting non HTTPS links etc.

One downside to using a library for this, is if you want to use a newer HTML elements, such as details and summary, the library may not yet know about them! This is currently the case with HTML Purifier and these tags. Most browsers support the, with IE and Edge being the main exceptions.

These tags allow a really simple pure HTML only way of having an expandable section on your page. Here’s an example.

Now that we know what these tags elements are, let’s look at how to implement them in HTML Purifier.

$config = \HTMLPurifier_Config::createDefault();
# Other configuration options you may use
$def = $config->getHTMLDefinition(true);
$def->addElement(
    'details',
    'Block',
    'Flow',
    'Common',
    array(
        'open' => new \HTMLPurifier_AttrDef_HTML_Bool(true)
    )
);
$def->addElement('summary', 'Inline', 'Inline', 'Common');

It’s not super-obvious when reading their docs, at least it wasn’t for myself, but you need to use new \HTMLPurifier_AttrDef_HTML_Bool(true) for the 'open' element. Simply using 'open' => 'Bool' doesn’t work, neither does 'open' => new \HTMLPurifier_AttrDef_HTML_Bool().

Hopefully this will make it easier for someone else out there to add these super-cool elements into their list of allowed elements.

2017 Year in Review – What I’m reading!

2018-01-02T10:57:44+00:00

It’s a brand new year. Many people have set their new years resolutions, companies are starting new projects, and for those in Australia and other countries where school years match calendar years, school kids are getting ready to start another year at school. In many ways, this blog is one such “new thing” that’s being started.

A common theme at the end of a year, is for a person or organisation to review the previous year, to highlight some special things they found and to look at what they’re wanting to do in the future. Unfortunately for me, I can’t do that this year as I’ve only just started documenting these things in a nice manner. However, I can still talk about things I want to do in the year to come!

Year to come

For 2018, there are three main things I’d like to do with this site.

Publish more consistently
Releasing guides and manuals
Give regular “What I’m reading” posts
Release tools online

Publish more consistently

If I get the other three done, this will be pretty easy. Basically though, I’ve ran other blogs in the past and I find that starting a blog is easy. Maintaining a blog is the hard part! If I can at the very least do weekly posts, I’ll be very satisfied with myself. If I manage monthly though, I’ll still be happy.

Releasing guides and manuals

In a way, I feel my recent posts on Rendering JSON Strings and Django/Axios/CSRF were kind of a lead up to this, but I’m wanting to do more in-depth guides for people to use in the future. I’ve found other people’s guides out there super-useful, but I often find that I have to take some aspects from multiple guides and combine them together to get exactly the information I need. I plan on finding a scenario that I’m currently working on tackling, and writing guides that look at the issue and how I’ve solved them. I have one in the works already, hopefully someone will find that useful!

What I’m reading

This isn’t something I see very often in tech related blogs, but I’ve found these to be interesting posts in other fields before. Basically, I’ll keep a list of articles I find useful, and at the end of the month publish that list so that other people can see what I’ve been finding useful. Hopefully this helps other people find out about other bloggers that they may not have heard of before, and may find an article they otherwise would have missed. I’ve started this with a “Recommended reading” page that lists other bloggers, but these posts will be linking to specific articles/posts, not just their main webpage.

Release tools online

We all have various websites that have useful tools that we go to in our day-to-day work, without which we’d have a harder time getting our jobs done. Two such tools I find super-useful are whatsmydns and whatsmybrowser. I’ve got some tools that I’ve personally been using, that I’d like to share publicly as I feel others would also find them useful. Developing these tools also serves as a great learning experience for me too!

2017 Recommended Reading – Year in review

Now that I’ve said what I want to do more of in the year to come, let’s start off number 3 in that list and do my first “What I’m reading” post and do something similar to a “Year in review” post at the same time?

Below are some articles that were published in 2017 that I found worth sharing.

Troy Hunt

I find it interesting that I’ve been following Troy for close to two years now. Boy how time flies…

Firstly, this series on Fixing Data Breaches by Troy Hunt was an interesting read, looking at ways that organisations can help minimise the damage when a data breach has occurred, ways of preventing them and how to encourage companies to do the right thing. This post may not be the most useful for individuals, but definitely worth the read if you’re a decision maker in a company. For ease of use, here’s the link to each individual part.

Following from this, albeit not chronologically, was a post Troy made several months earlier on how an organisation should/shouldn’t act after a data breach has occurred.

Other articles I’d include in a “Highlights of Troy Hunt for 2017” mix tape:

6 Step Happy Path to HTTPS
The one valuable things all websites have, and why Phishers want it
The trouble with politicians sharing passwords (or anyone for that matter!)

Scott Helme

Scott I’ve only been following for a shorter period of time, so it made this list a bit easier to write. Also he doesn’t post as much as Troy so there’s less articles to pick from, but they’re all equally useful.

Malware Hunting with CSP – This one I found particularly interesting. In my day-job hunting bugs in our software, I’m always looking for more ways of getting more useful debugging information!
Are EV certificates worth the paper they’re written on?
Sarahah – Particularly topical as a lot of my friends started using this around the time this article came out!
Let’s Encrypt with DNS Round Robin – An interesting read on using Let’s Encrypt on load balanced servers
nomx: The worlds most “secure” communications protocol followed by the comments on Hacker News, a ZDnet article on the topic and of course the comments on r/netsec

Others

I kept Troy’s and Scott’s in their own section, simply because they have so many. They’re not the only sites I read though! Other great articles to read:

Here’s hoping 2018 has even more worthwhile reads than 2017 did!

Rendering a JSON string inside a script with Django

2017-12-15T21:03:42+00:00

Building a list of form submissions in Django last night was causing me some issues. This is likely a common issue for people and something most people know already, but I wasn’t able to find a nice answer online after my searching, so here’s my solution for others to benefit from.

My scenario:

Build a list of submissions inside the view
Pass these into the template, so that Vue could render the details

I could have made the decision to load the page first, then use Axios to get the list of submissions, but I decided aginst that for now.

As such, I had my view setup along these lines:

from django.shortcuts import render
import json
from myapp.models import MyModel

def my_view(request):
    my_models = myModel.objects.all()
    model_list = []
    for model in my_models:
        new_model = {
            'id': model.id,
            'name': model.name,
            'picture': model.picture
        }
        model_list.append(new_model)

    context={
        'models': json.dumps(model_list)
    }

    return render(request, 'my_template.html')

(I would like to use model_to_dict from django.forms.models here but UUIDs don’t work nicely in my scenario…)

My Vue JS:

var models= new Vue(
    {
        el: '#models',
        delimiters: ["[%", "%]"],
        data: {
            models: model_list
        }
    }
);

End of course, the template:

The problem here was though, Django was escaping the JSON string automatically. After doing some googling I found many solutions that said to use the escapejs flag in conjunction with JSON.parse() but I wasn’t satisfied with that.

Django has a tag for disabling the auto-escaping it does though. This allows you to do the following:

<script>

</script>

And voila! Django renders the JSON String in such a way that Javascript can work with it right away, without having to use JSON.parse on it.

Django, Axios and CSRF

2017-11-20T21:05:19+00:00

I am currently building a simple Django based web-app. Part of this app is a registration form. It has a rather nice UI, built from Vue, and I’m POSTing the data back to Django using Axios.

One great thing about Django is it’s built-in CSRF protection. While I’m not an expert and can’t say it’s the best CSRF protection, at the very least it’s nice that it has something built in.

When posting via Axios though, you need to send through the CSRF token, somehow. Everyone was saying you had to post it back, but it took me a while to figure out where you should be placing it. Eventually I found this snippet from the official Django Docs.

The CSRF header name is HTTP_X_CSRFTOKEN by default, but you can customize it using the CSRF_HEADER_NAME setting.

This answered my questions, and I ended up with this little bit of JS inside my Vue method.

submit: function() {
    data: {...}; // The exact data doesn't matter
    csrftoken = Cookies.get('csrftoken'); // Using JS Cookies library
    headers = {X_CSRFTOKEN: csrftoken};
    axios.post(url,data,{headers: headers});
}

Now of course I had a .then and .catch section after axios’ .post, but that should at least give you some idea of how you can do it.

Update:

After reviewing this and how Django works, I’ve made a quick change to this document. Django adds “HTTP_” to all header names, and converts all dashes to underscores. So if your client sends a 'X-XSRF-TOKEN' header, the setting should be 'HTTP_X_XSRF_TOKEN'. In order to make use of the default setting, you need to send through X_CSRFTOKEN or X-CSRFTOKEN as the header name. If you use X-CSRFTOKEN you need to wrap it in ‘ in the JS code. IE:

headers = {'X-CSRFTOKEN': csrftoken};

Now you can avoid traps that I fell into.

Getting setup.py to add commands to the path

2017-01-09T21:00:37+00:00

There’s a number of articles out there that mention this, but they’re a bit of a pain to follow. Of them all, I found this set of docs to be the most useful, but still lacking.

This may, or may not, have been the case if I had read the entire docs, but that’s ok. This blog post is mainly designed just as a reference, for people who want to get their answers quickly, as I did, and without trial and error!

For those that want a quick answer, when you’re using setuptools in your setup function, you want to make use the of the entry_points keyword.

From the example above:

setup(
    ...
    entry_points = {
        'console_scripts': ['funniest-joke=funniest.command_line:main'],
    }
    ...
)

As a quick explanation of how this works:

funniest-joke – This is the command that you’ll enter into your terminal window
funniest.command_line:main – This tells setuptools which file, and which function to call. In this example, it will go into the funniest module, load the command_line file, and execute the main function

Deleting files within a directory with a given extension

2015-07-04T20:13:12+00:00

I wanted to delete a large number of .SRT files from some TV shows I had. (Sub-title files)

Going through each folder + selecting the files individually is tedious.

I wrote this bit of code, with the help of this Stack Overflow question.

Save this to the directory you want to delete the files from.

python remove_srt.py extension1 extension2

For example, to remove all JPG, TXT and PNG files:

python remove_srt.py jpg txt png

Script is here:

import sys
import os

num_deleted = 0
num_skipped = 0

def scandirs(path, extensions):
    global num_deleted
    global num_skipped
    for root, dirs, files in os.walk(path):
        for currentFile in files:
            print("processing file: " + currentFile)
            if any(currentFile.lower().endswith(ext) for ext in extensions):
                os.remove(os.path.join(root, currentFile))
                print("{} deleted".format(os.path.join(root, currentFile)))
                num_deleted += 1
            else:
                num_skipped += 1


if __name__ == '__main__':
    extensions = []
    for arg in sys.argv[1:]:
        
        extensions.append(arg)
    path = os.getcwd()
    scandirs(path, extensions)
    print("{} files deleted".format(num_deleted))
    print("{} files skipped".format(num_skipped))

Dec 2017 update

Oh boy, how time flies and how we learn. I’ll get a post up in early 2018 with a better way of doing this.