Nicholas de Jong

Hibp-downloader - download api.pwnedpasswords.com fast

2023-11-12T00:00:00+00:00

hibp-downloader is a CLI tool to efficiently download a local copy of the pwned password hash data from the very awesome HIBP pwned passwords api-endpoint using all the good bits; multiprocessing, async-processes, local-caching, content-etags and http2-connection pooling to make things as fast as is Pythonly possible.

Features

Easily resume interrupted download operations into a –data-path without re-clobbering api-source.
Only download hash-prefix content blocks when the source content has changed (via content ETAG values); thus making it easy to periodically re-sync when needed.
Ability to directly query for compromised password values from the data in-place; efficient enough to attach a service with reasonable loads.
Ability to generate a single text file with in-order pwned password hash values, similar to PwnedPasswordsDownloader from the HIBP team.
Per prefix file metadata in JSON format for easy data reuse by other tooling if required.

Usage

Install

pip install [--upgrade] hibp-downloader

Documentation

Plenty more documentation and examples here -

hibp-downloader.readthedocs.io

Source

github.com/threatpatrols/hibp-downloader

Python Package

https://pypi.org/project/hibp-downloader/

Arpwitch - a modern arpwatch replacement

2022-03-28T00:00:00+00:00

Arpwitch is a modern arpwatch replacement with JSON formatted outputs and easy options to trigger exec commands when network changes are observed.

outputs are JSON which makes it nice-and-easy to pair with other tools
easy triggers to –exec system commands when new network hosts are seen.

Since performing an nmap on new network hosts is a useful thing, there is also a built-in call to nmap that will invoke nmap with relatively mild but useful settings - you can easily fall back to a regular –exec if you need to do something more advanced.

Install

pip install [--upgrade] arpwitch

Example

arpwitch --debug --nmap --datafile /tmp/arpwitch.dat | jq .

The example above, watches for new network hosts; invokes an nmap scan on them when they are discovered and saves results in XML format; saves the arpwitch datafile to file /tmp/arpwitch.dat (it’s JSON); pipes the output through JQ to make it look pretty.

NB: arpwitch requires root/sudo access in order to packet-capture from the network interface(s) - the rather awesome scapy Python library is used for packet capture.

Documentation

Plenty more documentation and examples here -

arpwitch.readthedocs.io/en/latest

Source

Have moved this repo from where I originally published it under verbnetworks (the laboratory)

github.com/ndejong/arpwitch

Python Package

pypi.org/project/arpwitch

Usage

usage: arpwitch [-h] [-f <datafile>] [-i <seconds>] [-req | -noreq | -allreq]
                [-rep | -norep | -allrep] [-e <command>] [-n] [-u <user>]
                [-q <address>] [-v] [-w] [-d]

A modern arpwatch replacement with JSON formatted outputs and easy options to
execute commands when network changes are observed.

optional arguments:
  -h, --help            show this help message and exit
  -req, --new-request   Select ARP request packet events that include new
                        ip/hw addresses not yet observed (DEFAULT).
  -noreq, --no-request  Ignore all ARP request packet events.
  -allreq, --all-request
                        Select all ARP request packet events regardless if
                        addresses have been previously observed.
  -rep, --new-reply     Select only reply packet events that include new ip/hw
                        addresses not yet observed (DEFAULT).
  -norep, --no-reply    Ignore all ARP reply packet events.
  -allrep, --all-reply  Select all ARP reply packet events regardless if the
                        addresses have been previously observed.

datafile arguments:
  -f <datafile>, --datafile <datafile>
                        The arpwitch datafile where ARP event data is stored
                        as a JSON formatted file (REQUIRED). The datafile is
                        also easy to manually query and inspect with external
                        tools such as `jq`
  -i <seconds>, --interval <seconds>
                        Interval seconds between writing to the datafile
                        (DEFAULT: 30)

ARP event command execution arguments:
  The following exec command substitutions are available: {IP}=ipv4-address,
  {HW}=hardware-address, {TS}=timestamp-utc, {ts}=timestamp-utc-short

  -e <command>, --exec <command>
                        Command line to exec on selected ARP events. Commands
                        are run async
  -n, --nmap            A hard coded convenience --exec that causes nmap to be
                        run against the IPv4 target with nmap-XML formatted
                        output written to the current-working-directory. This
                        option cannot be used in conjunction with --exec.
  -u <user>, --user <user>
                        User to exec commands with, if not set this will be
                        the same user context as arpwitch.

run-mode arguments:
  Switches that invoke run-modes other than ARP capture.

  -q <address>, --query <address>
                        Query the <datafile> for an IPv4 or HW address and
                        return results in JSON formatted output and exit.
  -v, --version         Return the arpwitch version and exit.
  -w, --witch           Supply one witch to <stdout> and exit.
  -d, --debug           Debug messages to stdout.

Azure-cli via Docker and .bash_aliases entry

2022-02-15T00:00:00+00:00

If you’re a little cautious around everything from Microsoft then using their Docker container is a decent approach to manage those concerns. Unfortunately, as tends to be the case with a lot of Microsoft documentation, it’s verbose and long but is somehow missing details that make things real-world useful.

Fixing that then -

Use the following .bash_aliases entry to wrap the az alias-command with something that invokes a Docker container locally.

alias az="docker run --rm --interactive --log-driver=none --attach stdin --attach stdout --attach stderr --volume ${HOME}/.azure:/root/.azure:rw --volume ${HOME}/.ssh:/root.ssh mcr.microsoft.com/azure-cli az"

The above uses long-form docker args to make it a little easier to understand what it all means.

Breaking that down into parts -

docker run – causes the container from mcr.microsoft.com/azure-cli to be run with the command az (at the end of the command line)
–rm – causes the container to be removed after the command has completed.
–interactive – keeps the Docker console connection open for stdin/stdout/stderr
–log-driver=none – prevent otherwise hidden container log messages
–attach – attach each of stdin, stdout and stderr; which is important for az since stderr and stdout otherwise gets collapsed onto one-another making it not-possible to pipe output in your local environment, such as when piping through jq (further detail below)
–volume ${HOME}/.azure:/root/.azure – mount your ~/.azure into the Docker container so the Azure authentication is persistent between docker invocations; you may want to consider placing this mount on a separately encrypted mount.
–volume ${HOME}/.ssh:/root.ssh – mount your ~/.ssh path into the Docker container.

With this alias in-place you’ll be invoking a (local) Docker container to handle the “dirty” Microsoft stuff.

Further notes:-

be sure to use –attach to pass through stdin, stdout and stderr else you will have problems when piping output into something like jq - the issue is that the azure-cli tooling writes command progress/status output to the terminal in a way that gets hidden by using line-feed characters - if you redirect the output from docker to file or string and then view that string you may miss the fact that there are line(s) not visible - if you strip the line-feeds by piping | tr -d ‘\r’ you’ll then uncover the azure-cli status messages at the head of string :( that’s the long story.

Elasticsearch Kibana CLI (eskbcli)

2021-04-04T00:00:00+00:00

ElasticSearch Kibana CLI (eskbcli) is another tool I’ve had in my back-pocket for a while and frequently rely on when working threat-response / threat-management cases where direct access to ElasticSearch is not easily possible.

This situation comes up a little more often than you’d think in security-operations situations because the ElasticSearch backend is often tucked away and made out-of-reach (as it should be).

In short - ElasticSearch Kibana CLI (eskbcli) provides a shell interface to query an ElasticSearch backend via the Kibana frontend.

ElasticSearch Kibana CLI makes it possible to copy-paste query expressions directly from the Kibana user-interface and then locally access very large sets of result data. This makes eskbcli useful in SecOps situations where the ability to rapidly move from a Kibana query to raw data is valued.

Configuration options are available to adjust http-headers so-as-to enable access to Kibana in situations that require complex user-authentication such as when Kibana is positioned behind an OAuth reverse proxy or other session-based authentication arrangements.

Install

pip install [--upgrade] elasticsearch-kibana-cli

Documentation

Plenty of documentation at ReadTheDocs -

elasticsearch-kibana-cli.readthedocs.io

Source

github.com/ndejong/elasticsearch_kibana_cli

Python Package

pypi.python.org/pypi/elasticsearch-kibana-cli

Usage

Usage: eskbcli [OPTIONS] COMMAND [ARGS]...

  ElasticSearch Kibana CLI (`eskbcli`) provides a shell interface to query
  an ElasticSearch backend via the Kibana frontend which is useful in
  situations where the ElasticSearch backend is not otherwise accessible.

  ElasticSearch Kibana CLI makes it possible to copy-paste query expressions
  directly from the Kibana user-interface and then easily access very large
  sets of result data.  This makes the `eskbcli` useful in SecOps situations
  where the ability to rapidly move from a Kibana query to raw data is
  valued.

  Configuration options are available to adjust http-headers so-as-to enable
  access to Kibana in situations that require complex user-authentication
  such as when Kibana exists behind an OAuth reverse proxy or other session-
  based authentication arrangement.

  Documentation available https://elasticsearch-kibana-cli.readthedocs.io

Options:
  -c, --config TEXT  Config file location; overrides ESKBCLI_CONFIG_FILENAME
                     environment var and the default ~/.eskbcli value.

  -v, --verbose      Verbose logging messages (debug level).
  -q, --quiet        Quiet mode, takes priority over --verbose
  --version          Show the version and exit.
  --help             Show this message and exit.

Commands:
  list     List the available eskbcli search names.
  search   Execute the named search configuration.
  show     Show the named eskbcli search configuration.
  summary  Summary report for search result datafile; use "-" to pipe stdin.

SolarEdge API interface

2020-12-17T00:00:00+00:00

Outside the usual realm of security-things, last week I published a SolarEdge Interface, a command-line and a Python module interface to interact with the SolarEdge API service that’s a decent improvement over the existing ones out there.

In short -

All (documented) SolarEdge API endpoints are implemented with multisite support for endpoints that provide multisite queries.
Response data for all endpoints are available as a Python-dict structure; a Pandas-DataFrame or; as raw-JSON.
The command-line interface output can be formatted as a CSV; as a Pandas style JSON structure or; as plain-JSON.
Timestamps can be returned as datetime values with their respective site timezones applied. Doing so is the default behaviour, however this can be disabled if required.
Configuration via environment variables or config file is possible, thus making it safer to manage your API key value(s).

Install

  pip install [--upgrade] solaredge-interface

Documentation

solaredge-interface.readthedocs.io

Source

Github - github.com/ndejong/solaredge-interface

Usage

Usage: solaredge-interface [OPTIONS] COMMAND [ARGS]...

  The solaredge-interface provides a command-line interface to interact with
  the Python SolarEdgeAPI module which itself calls the SolarEdge public API
  endpoints at https://monitoringapi.solaredge.com making it even easier to
  access your data from SolarEdge.

  Configuration can be achieved through command arguments, environment
  values or config file.

  Documentation available https://solaredge-interface.readthedocs.io

Options:
  -c, --config TEXT       Override default config ~/.solaredge-interface
  -f, --format TEXT       Output format; csv, json, pandas (default: json)
  -v, --verbose           Verbose logging messages (debug level).
  -q, --quiet             Quiet mode, with priority over --verbose
  -W, --disable-warnings  Disable Python warnings.
  --version               Show the version and exit.
  --help                  Show this message and exit.

Commands:
  accounts                     Get the accessible >sub< accounts.
  site_current_power_flow      Current power flow between all elements of...
  site_data_period             Sites(s) start_date and end_date of...
  site_details                 Get site details; name, location, status,...
  site_energy                  Site(s) energy measurements
  site_energy_details          Detailed site energy measurements from meters
  site_environmental_benefits  Environmental benefits based on site energy...
  site_equipment_change_log    Equipment component replacements ordered by...
  site_equipment_data          Get specific inverter data for a given...
  site_equipment_sensors       Sensors in the site and connections
  site_inventory               Inventory of SolarEdge equipment at the site
  site_meters                  Meter lifetime energy, metadata and...
  site_overview                Sites(s) overview data
  site_power                   Site(s) power measurements
  site_power_details           Detailed site power measurements from meters
  site_storage_data            Detailed storage information from batteries
  site_time_frame_energy       Site(s) total energy produced for a given...
  sites                        Get the list of accessible sites
  version_current              Current version in <major.minor.revision>...
  version_supported            Supported version numbers in...

Digital Multimeter interface

2020-12-03T00:00:00+00:00

Wrote a digital multimeter CLI tool (and Python module) to read an old Digitech QM1538 multimeter sold by Jaycar, Australia. This was a weekend-project that started with a scratchy old data-sheet that described the serial-protocol used by this thing. I’d not written a serial-protocol decoder before and the tool would make my digital-multimeter usable via Linux and accessible remotely via SSH which is what I really wanted in this case.

It turns out that the underlying chipset (Fortune FS9721) used by this multimeter is rather common among other digital multimeters too, so the CLI and accompanying Python module should work for a whole host of other digital multimeters out there -

Digitech - QM1538
Digitek - DT4000ZC
PCE - PCEDM32
Tecpel - DMM8062
TekPower - TP4000ZC
UniTrend - UT30A
UniTrend - UT30E
UniTrend - UT60E
Voltcraft - VC820
Voltcraft - VC840

The Python module is written to accommodate new digital-multimeter protocols. If you happen to implement another digital-multimeter protocol please do send a merge request - happy days.

Install

pip install [--upgrade] digital-multimeter

Documentation

digital-multimeter.readthedocs.io

Source

github.com/ndejong/digital-multimeter

Usage

Usage: dmm [OPTIONS] COMMAND [ARGS]...

  Digital multimeter CLI tool utilizing the Python3 DigitalMultimeter module.

  Configuration can be achieved through command arguments, environment values
  or config file.

  Documentation available https://digital-multimeter.readthedocs.io

Options:
  -q, --quiet             Quiet mode; priority over --verbose
  -v, --verbose           Verbose logging; debug level.
  -W, --disable-warnings  Disable Python warnings.
  --version               Show the version and exit.
  --help                  Show this message and exit.

Commands:
  models  Provides a list of the supported digital multimeter models
  read    Read the digital multimeter and output data in various formats

Env-Alias environment helper utility

2020-03-01T00:00:00+00:00

Env-Alias is a helper utility to create shell alias commands that easily set collections of environment variables often with secret values from a variety of data-sources and data-formats.

Typically this tool is invoked via an entry in .bash_aliases with an entry in the form

  eval $(env-alias my-alias-name ~/path-to/my-alias-name-config-file.yml)

Where this example would establish a shell alias command for the alias my-alias-name that then invokes the env-alias-generator with configuration from ~/path-to/my-alias-name-config-file.yml

That all might sound quirky, however the mechanism is enormously useful in working with large sets of environment variables from encrypted or otherwise secured data-sources which means an environment configuration can be easily committed to source control without the secret values.

Features

Data sources: local-file, http-remote, exec-stdout or directly-set
Source file formats: text, ini, json, yaml
Select using jq-style selectors, xpath selectors or line-numbers
Reference other env values within configuration
Content-Type detection for http-remote data-sources to automatically assign appropriate parser
Run exec helper commands without content assignment for creating setups
Logs output to STDERR
Easy installation using PyPI pip
Plenty of documentation and examples - https://env-alias.readthedocs.io

Install

pip install [--upgrade] env-alias

Documentation

Read the documentation, there really is a lot of awesome things to do with env-alias

env-alias.readthedocs.io

Source

github.com/ndejong/env-alias

Python client for pfSense FauxAPI

2019-03-17T00:00:00+00:00

For longest time I’d been meaning to roll a PyPi package for the Python client interface to FauxAPI, and it is now a thing - it also closes the long open issue #22 asking about PyPi from way back.

pfsense-fauxapi

Install it using pip:-

  pip install pfsense-fauxapi

Github project

github.com/ndejong/pfsense_fauxapi_client_python

Command Line

Additionally this pip-package provides a command-line interface to work with FauxAPI

usage: fauxapi [-h] [--host [host]] [--apikey [key]] [--apisecret [secret]]
               [--verified-ssl] [--debug]
               [function] [[function] ...] [[function-args]]

FauxAPI

optional arguments:
  -h, --help            show this help message and exit

Call:
  --host [host]         Host address of the target pfSense host with the
                        PfsenseFauxapi package installed.
  --apikey [key]        FauxAPI apikey value - alternatively via the
                        FAUXAPI_APIKEY environment variable.
  --apisecret [secret]  FauxAPI apisecret value - alternatively via the
                        FAUXAPI_APIKEY environment variable.
  --verified-ssl        Enable SSL certificate checks - default does NOT check
                        SSL certificates.
  --debug               Enable debug response from the remote FauxAPI -
                        helpful in tracking down issues.
  [function]            The FauxAPI function being called
  [function-args]       Arguments to the function, space separated

Command line example, using environment variables to pass the FAUXAPI_APIKEY and FAUXAPI_APIKEY credentials.

$ fauxapi --host 192.168.1.200 gateway_status | jq .
{
  "callid": "5c8d0f7361cba",
  "action": "gateway_status",
  "message": "ok",
  "data": {
    "gateway_status": {
      "10.11.12.1": {
        "monitorip": "10.10.10.1",
        "srcip": "10.10.10.200",
        "name": "WAN_DHCP",
        "delay": "0.422ms",
        "stddev": "0.073ms",
        "loss": "0.0%",
        "status": "none"
      }
    }
  }
}

Package Testing

Tests for (almost) all client-side function calls are implemented with mocked API response data you can check them with pytest

  python setup.py test

Packages are tested via Travis travis-ci.org/ndejong/pfsense_fauxapi_client_python

Package Build

Should you need/want to build the package from source

# obtain the source material
git clone https://github.com/ndejong/pfsense_fauxapi_client_python.git
cd pfsense_fauxapi_client_python

# build the package
python setup.py sdist bdist_wheel

# install the package from the .whl file in the dist path 
pip install dist/pfsense*.whl

Terraform + Digital Ocean Droplets

2019-02-20T00:00:00+00:00

This Terraform module creates a Digital Ocean Droplet using Terraform with desirable additional features. The module is essentially a wrapper around the Digital Ocean provider using a cloudinit script to provide additional features:-

remount existing volumes
create an initial user with an sshkey

Update

2022-03-28 - the source for this module has been migrated from github verbnetworks to github ndejong - the module at registry.terraform.io points to verbnetworks which is redirected to ndejong - yeeeee.

Current version is now v0.2.1

Source

github.com/ndejong/terraform-digitalocean-droplet

Terraform Module

registry.terraform.io/modules/ndejong/droplet/digitalocean

Keybase.io Network Graphs

2019-01-06T00:00:00+00:00

Recently I became interested in Keybase.io and really wanted to be able explore my network-graph using Gephi, so I wrote a quick tool to collect the data from Keybase; dump it into GraphML file; then loaded it up in Gephi. The results nicely highlight who-knows-who; the strengths and weaknesses in my own Keybase network; and super connector users with many followers.

Keybase

Keybase is an interesting project, and the circle-of-trust improvements their tooling provides is probably better than anything else out there at the moment. The obvious concern is whether they have a business model that provides sufficient value to people (or enterprises) that will make Keybase long-term viable.

Github project

https://github.com/ndejong/keybase-network-graph

Command line

$ ./keybase-network-graph.py -h
usage: keybase-network-graph.py [-h] --uid <uid> [--depth <depth>]
                                [--path <path>] [--nograph]
Keybase Network Grapher

optional arguments:
  -h, --help       show this help message and exit
  --uid <uid>      Initial uid to start collecting data on, required.
  --depth <depth>  Maximum depth of user connections to collect data on,
                   default = 1
  --path <path>    Maximum depth of user connections to collect data down to,
                   default = /tmp/keybase-network-graph
  --nograph        Prevent GraphML generation.

Nicholas de Jong

Hibp-downloader - download api.pwnedpasswords.com fast

Features

Usage

Install

Documentation

Source

Python Package

Arpwitch - a modern arpwatch replacement

Install

Example

Documentation

Source

Python Package

Usage

Azure-cli via Docker and .bash_aliases entry

Elasticsearch Kibana CLI (eskbcli)

Install

Documentation

Source

Python Package

Usage

SolarEdge API interface

Install

Documentation

Source

Usage

Digital Multimeter interface

Install

Documentation

Source

Usage

Env-Alias environment helper utility

Features

Install

Documentation

Source

Python client for pfSense FauxAPI

pfsense-fauxapi

Github project

Command Line

Package Testing

Package Build

Terraform + Digital Ocean Droplets

Update

Source

Terraform Module

Keybase.io Network Graphs

Keybase

Github project

Command line

Visualizations in Gephi