From 73dd080f91186217da592b89f91bc8077a6d87f8 Mon Sep 17 00:00:00 2001 From: Midgard Date: Tue, 22 Dec 2020 21:53:32 +0100 Subject: [PATCH] Initial commit --- LICENSE | 202 +++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 20 +++++ fetch_rates.py | 112 +++++++++++++++++++++++++++ rates | 34 +++++++++ update-rates | 41 ++++++++++ 5 files changed, 409 insertions(+) create mode 100644 LICENSE create mode 100644 README.md create mode 100755 fetch_rates.py create mode 100755 rates create mode 100755 update-rates diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..d645695 --- /dev/null +++ b/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/README.md b/README.md new file mode 100644 index 0000000..a2c7c3d --- /dev/null +++ b/README.md @@ -0,0 +1,20 @@ +This is a setup to automatically update a Ledger pricedb. It’s meant to work +for the author and is published for inspiration, not to be a tool that works +for everyone. + +------------------------------------------------------------------------------- + +Requires Python 3.6 or higher. + +Put `rates` and `fetch_rates.py` somewhere on a server. + +Put `update-rates` somewhere in your path on your local machine. + +Choose a location to put the serverside data files, e.g. `/var/local/pricedb`. +Configure all three scripts as appropriate. + +Create a cron job for `fetch_rates.py` on the server. + + 0 0 * * * /usr/local/bin/python3 /usr/local/lib/pricedb/fetch_rates.py + +To update the pricedb on your machine, run `update-rates`. diff --git a/fetch_rates.py b/fetch_rates.py new file mode 100755 index 0000000..13f528a --- /dev/null +++ b/fetch_rates.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 + +# Copyright 2020 Midgard +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Configure me +FUNDS = { + # To get these URLs, you'll need to go to the fund page and find the frame that has the actual + # data and take its URL + "PREFERRED_SYMBOL": "https://www.tijd.be/customers/mediafin.be/funds_tijd/123/Fund/456789?t=" +} +DIRECTORY = "/var/local/rate_fetcher" +# Currencies fetched are always all those that are published by the ECB, currently +# AUD, BGN, BRL, CAD, CHF, CNY, CZK, DKK, GBP, HKD, HRK, HUF, IDR, ILS, INR, ISK, +# JPY, KRW, MXN, MYR, NOK, NZD, PHP, PLN, RON, RUB, SEK, SGD, THB, TRY, USD, ZAR + + +import requests +import os.path +import re +from typing import NamedTuple + +class LogItem(NamedTuple): + symbol: str + date: str + value: str + + +def get_ecb(): + r = requests.get("https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml") + m = re.search(r"", r.text) + assert m + date = m.group(1) + + for item in re.findall(r"", r.text): + symbol, value = item + yield LogItem( + symbol=symbol, + date=date, + value="{:.5f}".format(1.0 / float(value)), + ) + + +def get_fund(symbol, url): + r = requests.get(url) + m = re.search(r"Actuele NIW op ([0-9]{1,2})/([0-9]{1,2})/([0-9]{4}).*?([0-9]+),([0-9]+)", r.text, re.MULTILINE|re.DOTALL) + assert m + groups = m.groups() + + return LogItem( + symbol=symbol, + date="{2}-{1:0>2}-{0:0>2}".format(*groups), + value="{3}.{4}".format(*groups), + ) + + +def get_funds(): + return [ + get_fund(symbol, url) + for symbol, url in FUNDS.items() + ] + + +START_OF_STREAM = 0 +CURRENT_POSITION = 1 +END_OF_STREAM = 2 + +def last_line(fh, max_line_length=80): + """Simple and stupid way to read the last line of a file""" + try: + fh.seek(-max_line_length, END_OF_STREAM) + except OSError: + fh.seek(0, START_OF_STREAM) + lines = fh.readlines() + return lines[-1] if lines else None + + +def format_logline(entry: LogItem): + return f"P {entry.date} 18:00:00 {entry.symbol} € {entry.value}" + + +def log(entry: LogItem): + filename = os.path.join(DIRECTORY, "{symbol}.log".format(symbol=entry.symbol)) + try: + with open(filename, "rb") as fh: + if entry.date.encode() in (last_line(fh) or ""): + return + except FileNotFoundError: + pass + with open(filename, "a") as fh: + print(format_logline(entry), file=fh) + + +def main(): + for entry in [*get_ecb(), *get_funds()]: + log(entry) + + +if __name__ == '__main__': + main() diff --git a/rates b/rates new file mode 100755 index 0000000..5eb298c --- /dev/null +++ b/rates @@ -0,0 +1,34 @@ +#!/bin/bash + +# Copyright 2020 Midgard +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Configure me +cd /var/local/rate_fetcher + + +date="$1" +shift 1 +while [ -n "$1" ]; do + tail -n100 < "$1.log" + shift 1 +done | sort | \ + python3 -c " +import sys +date = '$date' +for line in sys.stdin: + if line[2:2 + len(date)] > date: + sys.stdout.write(line) +" diff --git a/update-rates b/update-rates new file mode 100755 index 0000000..a3e9f46 --- /dev/null +++ b/update-rates @@ -0,0 +1,41 @@ +#!/bin/bash + +# Copyright 2020 Midgard +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Configure me +SSHARGS="myserver.example.org" +SYMBOLS="USD GBP" +SERVERSIDE_PATH_TO_RATES="/usr/local/lib/pricedb/rates" + + +set -euo pipefail + +pricedb="$HOME/ledger/pricedb.ledger" +cd "$(dirname "$pricedb")" + +last_date="$(tail -n1 "$pricedb" | sed 's/P \([0-9 :-]*\) .*/\1/')" +printf 'Fetching all exchange rates after %s\n' "$last_date" + +values="$(ssh $SSHARGS "$SERVERSIDE_PATH_TO_RATES '$last_date' $SYMBOLS")" +if [ $? -eq 0 -a -n "$values" ]; then + printf '%s\n' "$values" >> "$pricedb" + printf '%s\n' "$values" +fi + +git diff --quiet "$pricedb" && exit +git diff --cached --quiet || { echo "Not adding to Git because there are staged changes"; exit; } +git add "$pricedb" +git commit -m "Update pricedb"