Planet Debian

Subscribe to Planet Debian feed
Planet Debian - https://planet.debian.org/
Updated: 2 hours 51 min ago

Matthew Garrett: Initial thoughts on MongoDB's new Server Side Public License

8 hours 14 min ago
MongoDB just announced that they were relicensing under their new Server Side Public License. This is basically the Affero GPL except with section 13 largely replaced with new text, as follows:

If you make the functionality of the Program or a modified version available to third parties as a service, you must make the Service Source Code available via network download to everyone at no charge, under the terms of this License. Making the functionality of the Program or modified version available to third parties as a service includes, without limitation, enabling third parties to interact with the functionality of the Program or modified version remotely through a computer network, offering a service the value of which entirely or primarily derives from the value of the Program or modified version, or offering a service that accomplishes for users the primary purpose of the Software or modified version.

“Service Source Code” means the Corresponding Source for the Program or the modified version, and the Corresponding Source for all programs that you use to make the Program or modified version available as a service, including, without limitation, management software, user interfaces, application program interfaces, automation software, monitoring software, backup software, storage software and hosting software, all such that a user could run an instance of the service using the Service Source Code you make available.

MongoDB admit that this license is not currently open source in the sense of being approved by the Open Source Initiative, but say:We believe that the SSPL meets the standards for an open source license and are working to have it approved by the OSI.

At the broadest level, AGPL requires you to distribute the source code to the AGPLed work[1] while the SSPL requires you to distribute the source code to everything involved in providing the service. Having a license place requirements around things that aren't derived works of the covered code is unusual but not entirely unheard of - the GPL requires you to provide build scripts even if they're not strictly derived works, and you could probably make an argument that the anti-Tivoisation provisions of GPL3 fall into this category.

A stranger point is that you're required to provide all of this under the terms of the SSPL. If you have any code in your stack that can't be released under those terms then it's literally impossible for you to comply with this license. I'm not a lawyer, so I'll leave it up to them to figure out whether this means you're now only allowed to deploy MongoDB on BSD because the license would require you to relicense Linux away from the GPL. This feels sloppy rather than deliberate, but if it is deliberate then it's a massively greater reach than any existing copyleft license.

You can definitely make arguments that this is just a maximalist copyleft license, the AGPL taken to extreme, and therefore it fits the open source criteria. But there's a point where something is so far from the previously accepted scenarios that it's actually something different, and should be examined as a new category rather than already approved categories. I suspect that this license has been written to conform to a strict reading of the Open Source Definition, and that any attempt by OSI to declare it as not being open source will receive pushback. But definitions don't exist to be weaponised against the communities that they seek to protect, and a license that has overly onerous terms should be rejected even if that means changing the definition.

In general I am strongly in favour of licenses ensuring that users have the freedom to take advantage of modifications that people have made to free software, and I'm a fan of the AGPL. But my initial feeling is that this license is a deliberate attempt to make it practically impossible to take advantage of the freedoms that the license nominally grants, and this impression is strengthened by it being something that's been announced with immediate effect rather than something that's been developed with community input. I think there's a bunch of worthwhile discussion to have about whether the AGPL is strong and clear enough to achieve its goals, but I don't think that this SSPL is the answer to that - and I lean towards thinking that it's not a good faith attempt to produce a usable open source license.

(It should go without saying that this is my personal opinion as a member of the free software community, and not that of my employer)

[1] There's some complexities around GPL3 code that's incorporated into the AGPLed work, but if it's not part of the AGPLed work then it's not covered

comments

Reproducible builds folks: Reproducible Builds: Weekly report #181

10 hours 35 min ago

Here’s what happened in the Reproducible Builds effort between Sunday October 7 and Saturday October 13 2018:

Another brief reminder that another Reproducible Builds summit will be taking place between 11th—13th December 2018 in Mozilla’s offices in Paris. If you are interested in attending please send an email to holger@layer-acht.org. More details can also be found on the corresponding event page of our website.

diffoscope development

diffoscope is our in-depth “diff-on-steroids” utility which helps us diagnose reproducibility issues in packages) was updated this week, including contributions from:

Packages reviewed and fixed, and bugs filed Test framework development

There were a large number of updates to our Jenkins-based testing framework that powers tests.reproducible-builds.org by Holger Levsen this month, including:

In addition, Mattia Rizzolo performed some node administration (1, 2).

Misc.

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Julien Danjou: More GitHub workflow automation

16 October, 2018 - 19:39

The more you use computers, the more you see the potentials for automating everything. Who doesn't love that? By building Mergify those last months, we've decided it was time bring more automation to the development workflow.

Mergify's first version was a minimal viable product around automating the merge of pull requests. As I wrote a few months ago, we wanted to automate the merge of pull requests when it was ready to be merged. For most projects, this is easy and consists of a simple rule: "it must be approved by a developer and pass the CI".

Evolving on Feedback

For the first few months, we received a lot of feedback from our users. They were enthusiastic about the product but were frustrated by a couple of things.

First, Mergify would mess up with branch protections. We thought that people wanted the GitHub UI to match their rules. As I'll explain later, it turns out to be only partially true, and we found a workaround.

Then, Mergify's abilities were capped by some of the limitations of the GitHub workflow and API. For example, GitHub would only allow rules per branch, whereas our users wanted to have rules applied based on a lot of different criteria.

Building the Next Engine

We rolled up our sleeves and started to build that new engine. The first thing was to get rid of the GitHub branch protection feature altogether and leveraging the Checks API to render something useful to the users in the UI. You can now have a complete overview of the rules that will be applied to your pull requests in the UI, making it easy to understand what's happening.

Then, we wrote a new matching engine that would allow matching any pull requests based on any of its attributes. You can now automate your workflow with a finer-grained configuration.

What Does It Look Like?

Here's a simple rule you could write:

pull_request_rules:
  - name: automatic merge on approval and CI pass
    conditions:
     - "#approved-reviews-by>=1"
     - status-success=continuous-integration/travis-ci/pr
     - label!=work-in-progress
    actions:
      merge:
        method: merge

With that, any pull request that has been approved by a collaborator, passes the Travis CI job and does not have the label work-in-progress will be automatically merged by Mergify.

You could use even more actions to backport this pull request to another branch, close the pull request or add/remove labels. We're starting users to build amazing workflow with that engine!

We're thrilled by this new version we launched this week and glad we're getting amazing feedback (again) for our users.

When you give it a try, drop me a note and let me know what you think about this!

Michal Čihař: uTidylib 0.4

15 October, 2018 - 21:30

Two years ago, I've taken over uTidylib maintainership. Two years has passed without any bigger contribution, but today there is a new version with support for recent html-tidy and Python 3.

The release still can't be uploaded to PyPI (see https://github.com/pypa/warehouse/issues/4860), but it's available for download from my website or tagged in the GitHub repository.

Full list of changes is quite small:

  • Compatibility with html-tidy 5.6.0.
  • Added support for Python 3.

Anyway as I can not update PyPI entry, the downloads are currently available only on my website: https://cihar.com/software/utidylib/

Filed under: Debian English SUSE uTidylib

Robert McQueen: Flatpaks, sandboxes and security

15 October, 2018 - 20:40

Last week the Flatpak community woke to the “news” that we are making the world a less secure place and we need to rethink what we’re doing. Personally, I’m not sure this is a fair assessment of the situation. The “tl;dr” summary is: Flatpak confers many benefits besides the sandboxing, and even looking just at the sandboxing, improving app security is a huge problem space and so is a work in progress across multiple upstream projects. Much of what has been achieved so far already delivers incremental improvements in security, and we’re making solid progress on the wider app distribution and portability problem space.

Sandboxing, like security in general, isn’t a binary thing – you can’t just say because you have a sandbox, you have 100% security. Like having two locks on your front door, two front doors, or locks on your windows too, sensible security is about defense in depth. Each barrier that you implement precludes some invalid or possibly malicious behaviour. You hope that in total, all of these barriers would prevent anything bad, but you can never really guarantee this – it’s about multiplying together probabilities to get a smaller number. A computer which is switched off, in a locked faraday cage, with no connectivity, is perfectly secure – but it’s also perfectly useless because you cannot actually use it. Sandboxing is very much the same – whilst you could easily take systemd-nspawn, Docker or any other container technology of choice and 100% lock down a desktop app, you wouldn’t be able to interact with it at all.

Network services have incubated and driven most of the container usage on Linux up until now but they are fundamentally different to desktop applications. For services you can write a simple list of permissions like, “listen on this network port” and “save files over here” whereas desktop applications have a much larger number of touchpoints to the outside world which the user expects and requires for normal functionality. Just thinking off the top of my head you need to consider access to the filesystem, display server, input devices, notifications, IPC, accessibility, fonts, themes, configuration, audio playback and capture, video playback, screen sharing, GPU hardware, printing, app launching, removable media, and joysticks. Without making holes in the sandbox to allow access to these in to your app, it either wouldn’t work at all, or it wouldn’t work in the way that people have come to expect.

What Flatpak brings to this is understanding of the specific desktop app problem space – most of what I listed above is to a greater or lesser extent understood by Flatpak, or support is planned. The Flatpak sandbox is very configurable, allowing the application author to specify which of these resources they need access to. The Flatpak CLI asks the user about these during installation, and we provide the flatpak override command to allow the user to add or remove these sandbox escapes. Flatpak has introduced portals into the Linux desktop ecosystem, which we’re really pleased to be sharing with snap since earlier this year, to provide runtime access to resources outside the sandbox based on policy and user consent. For instance, document access, app launching, input methods and recursive sandboxing (“sandbox me harder”) have portals.

The starting security position on the desktop was quite terrible – anything in your session had basically complete access to everything belonging to your user, and many places to hide.

  • Access to the X socket allows arbitrary input and output to any other app on your desktop, but without it, no app on an X desktop would work. Wayland fixes this, so Flatpak has a fallback setting to allow Wayland to be used if present, and the X socket to be shared if not.
  • Unrestricted access to the PulseAudio socket allows you to reconfigure audio routing, capture microphone input, etc. To ensure user consent we need a portal to control this, where by default you can play audio back but device access needs consent and work is under way to create this portal.
  • Access to the webcam device node means an app can capture video whenever it wants – solving this required a whole new project.
  • Sandboxing access to configuration in dconf is a priority for the project right now, after the 1.0 release.

Even with these caveats, Flatpak brings a bunch of default sandboxing – IPC filtering, a new filesystem, process and UID namespace, seccomp filtering, an immutable /usr and /app – and each of these is already a barrier to certain attacks.

Looking at the specific concerns raised:

  • Hopefully from the above it’s clear that sandboxing desktop apps isn’t just a switch we can flick overnight, but what we already have is far better than having nothing at all. It’s not the intention of Flatpak to somehow mislead people that sandboxed means somehow impervious to all known security issues and can access nothing whatsoever, but we do want to encourage the use of the new technology so that we can work together on driving adoption and making improvements together. The idea is that over time, as the portals are filled out to cover the majority of the interfaces described, and supported in the major widget sets / frameworks, the criteria for earning a nice “sandboxed” badge or submitting your app to Flathub will become stricter. Many of the apps that access --filesystem=home are because they use old widget sets like Gtk2+ and frameworks like Electron that don’t support portals (yet!). Contributions to improve portal integration into other frameworks and desktops are very welcome and as mentioned above will also improve integration and security in other systems that use portals, such as snap.
  • As Alex has already blogged, the freedesktop.org 1.6 runtime was something we threw together because we needed something distro agnostic to actually be able to bootstrap the entire concept of Flatpak and runtimes. A confusing mishmash of Yocto with flatpak-builder, it’s thankfully nearing some form of retirement after a recent round of security fixes. The replacement freedesktop-sdk project has just released its first stable 18.08 release, and rather than “one or two people in their spare time because something like this needs to exist”, is backed by a team from Codethink and with support from the Flatpak, GNOME and KDE communities.
  • I’m not sure how fixing and disclosing a security problem in a relatively immature pre-1.0 program (in June 2017, Flathub had less than 50 apps) is considered an ongoing problem from a security perspective. The wording in the release notes?

Zooming out a little bit, I think it’s worth also highlighting some of the other reasons why Flatpak exists at all – these are far bigger problems with the Linux desktop ecosystem than app security alone, and Flatpak brings a huge array of benefits to the table:

  • Allowing apps to become agnostic of their underlying distribution. The reason that runtimes exist at all is so that apps can specify the ABI and dependencies that they need, and you can run it on whatever distro you want. Flatpak has had this from day one, and it’s been hugely reliable because the sandboxed /usr means the app can rely on getting whatever they need. This is the foundation on which everything else is built.
  • Separating the release/update cadence of distributions from the apps. The flip side of this, which I think is huge for more conservative platforms like Debian or enterprise distributions which don’t want to break their ABIs, hardware support or other guarantees, is that you can still get new apps into users hands. Wider than this, I think it allows us huge new freedoms to move in a direction of reinventing the distro – once you start to pull the gnarly complexity of apps and their dependencies into sandboxes, your constraints are hugely reduced and you can slim down or radically rethink the host system underneath. At Endless OS, Flatpak literally changed the structure of our engineering team, and for the first time allowed us to develop and deliver our OS, SDK and apps in independent teams each with their own cadence.
  • Disintermediating app developers from their users. Flathub now offers over 400 apps, and (at a rough count by Nick Richards over the summer) over half of them are directly maintained by or maintained in conjunction with the upstream developers. This is fantastic – we get the releases when they come out, the developers can choose the dependencies and configuration they need – and they get to deliver this same experience to everyone.
  • Decentralised. Anyone can set up a Flatpak repo! We started our own at Flathub because there needs to be a center of gravity and a complete story to build out a user and developer base, but the idea is that anyone can use the same tools that we do, and publish whatever/wherever they want. GNOME uses GitLab CI to publish nightly Flatpak builds, KDE is setting up the same in their infrastructure, and Fedora is working on completely different infrastructure to build and deliver their packaged applications as Flatpaks.
  • Easy to build. I’ve worked on Debian packages, RPMs, Yocto, etc and I can honestly say that flatpak-builder has done a very good job of making it really easy to put your app manifest together. Because the builds are sandboxed and each runtimes brings with it a consistent SDK environment, they are very reliably reproducible. It’s worth just calling this out because when you’re trying to attract developers to your platform or contributors to your app, hurdles like complex or fragile tools and build processes to learn and debug all add resistance and drag, and discourage contributions. GNOME Builder can take any flatpak’d app and build it for you automatically, ready to hack within minutes.
  • Different ways to distribute apps. Using OSTree under the hood, Flatpak supports single-file app .bundles, pulling from OSTree repos and OCI registries, and at Endless we’ve been working on peer-to-peer distribution like USB sticks and LAN sharing.

Nobody is trying to claim that Flatpak solves all of the problems at once, or that what we have is anywhere near perfect or completely secure, but I think what we have is pretty damn cool (I just wish we’d had it 10 years ago!). Even just in the security space, the overall effort we need is huge, but this is a journey that we are happy to be embarking together with the whole Linux desktop community. Thanks for reading, trying it out, and lending us a hand.

Lars Wirzenius: Rewrote summain from Python to Rust

15 October, 2018 - 15:13

I've been learning Rust lately. As part of that, I rewrote my summain program from Python to Rust (see summainrs). It's not quite a 1:1 rewrite: the Python version outputs RFC822-style records, the Rust one uses YAML. The Rust version is my first attempt at using multithreading, something I never added to the Python version.

Results:

  • Input is a directory tree with 8.9 gigabytes of data in 9650 files and directories.
  • Each file gets stat'd, and regular files get SHA256 computed.
  • Run on a Thinkpad X220 laptop with a rotating hard disk. Two CPU cores, 4 hyperthreads. Mostly idle, but desktop-py things running in the background. (Not a very systematic benchmark.)
  • Python version: 123 seconds wall clock time, 54 seconds user, 6 second system time.
  • Rust version: 61 seconds wall clock (50% of the speed), 56 seconds user (104%), and 4 seconds system time (67&).

A nice speed improvement, I think. Especially, since the difference between the single and multithreaded version of the Rust program is four characters (par_iter instead of iter in the process_chunk function).

Louis-Philippe Véronneau: A Good Harvest of the Devil's Lettuce

15 October, 2018 - 10:30

You might have heard that Canada's legalising marijuana in 2 days. Even though I think it's a pretty good idea, this post is not about pot, but about another type of Devil's Lettuce: hops.

As we all know, homebrewing beer is a gateway into growing hops, a highly suspicious activity that attracts only marginals and deviants. Happy to say I've been successfully growing hops for two years now and this year's harvest has been bountiful.

Two years ago, I planted two hops plants, one chinook and one triple pearl. A year prior to this I had tried to grow a cascade plant in a container on my balcony, but it didn't work out well. This time I got around to plant the rhizomes in the ground under my balcony and had the bines grow on ropes.

Although I've been having trouble with the triple pearl (the soil where I live is thick and heavy clay - not the best for hops), the chinook has been growing pretty well.

Harvest time is always fun and before taking the bines down, I didn't know how much cones I would get this year. I'd say compared to last year, I tripled my yield. With some luck (and better soil), I should be able to get my triple pearl to produce cones next year.

Here a nice poem about the usefulness of hops written by Thomas Tusser in 1557:

      The hop for his profit I thus do exalt,
      It strengtheneth drink and it flavoureth malt;
      And being well-brewed long kept it will last,
      And drawing abide, if ye draw not too fast.

So remember kids, don't drink and upload and if you decide to grow some of the Devil's Lettuce, make sure you use it to flavoureth malt and not your joint. The ones waging war on drugs might not like it.

Dirk Eddelbuettel: RcppCCTZ 0.2.5

15 October, 2018 - 06:04

A new bugfix release 0.2.5 of RcppCCTZ got onto CRAN this morning – just a good week after the previous release.

RcppCCTZ uses Rcpp to bring CCTZ to R. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. And while CCTZ is made by Google(rs), it is not an official Google product. The RcppCCTZ page has a few usage examples and details. This package was the first CRAN package to use CCTZ; by now at least three others do—but decided in their infinite wisdom to copy the sources yet again into their packages. Sigh.

This version corrects two bugs. We were not properly accounting for those poor systems that do not natively have nanosecond resolution. And I missed a feature in the Rcpp DatetimeVector class by not setting the timezone on newly created variables; this too has been fixed.

Changes in version 0.2.5 (2018-10-14)
  • Parsing to Datetime was corrected on systems that do not have nanosecond support in C++11 chrono (#28).

  • DatetimeVector objects are now created with their timezone attribute when available.

  • The toTz functions is now vectorized (#29).

  • More unit tests were added, and some conditioning on Solaris (mostly due to missing timezone info) was removed.

We also have a diff to the previous version thanks to CRANberries. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Jeremy Bicha: Google Cloud Print in Ubuntu

14 October, 2018 - 21:31

There is an interesting hidden feature available in Ubuntu 18.04 LTS and newer. To enable this feature, first install cpdb-backend-gcp.

sudo apt install cpdb-backend-gcp

Make sure you are signed in to Google with GNOME Online Accounts. Open the Settings app1gnome-control-center to the Online Accounts page. If your Google account is near the top above the Add an account section, then you’re all set.

Now when you print a document, there will be an extra Save to Google Drive option. If you have any Google Cloud Print printers, they should show up here too. Saving to Google Drive saves a PDF of your document to your Google Drive account.

This feature was developed by Nilanjana Lodh and Abhijeet Dubey when they were Google Summer of Code 2007 participants. Their mentors were Till Kampetter, Aveek Basu, and Felipe Borges.

Till has been trying to get this feature installed by default in Ubuntu since 18.04 LTS, but it looks like it won’t make it in until 19.04.

I haven’t seen this feature packaged in any other Linux distros yet. That might be because people don’t know about this feature so that’s why I’m posting about it today! If you are a distro packager, the 3 packages you need are cpdb-libs , cpdb-backend-gcp, and cpdb-backend-cups. The final package enables easy printing to any IPP printer. (I didn’t mention it earlier because I believe Ubuntu 18.04 LTS already supports that feature through a different package.)

Julian Andres Klode: The demise of G+ and return to blogging (w/ mastodon integration)

14 October, 2018 - 04:03

I’m back to blogging, after shutting down my wordpress.com hosted blog in spring. This time, fully privacy aware, self hosted, and integrated with mastodon.

Let’s talk details: In spring, I shutdown my wordpress.com hosted blog, due to concerns about GDPR implications with comment hosting and ads and stuff. I’d like to apologize for using that, back when I did this (in 2007), it was the easiest way to get into blogging. Please forgive me for subjecting you to that!

Recently, Google announced the end of Google+. As some of you might know, I posted a lot of medium-long posts there, rather than doing blog posts; especially after I disabled the wordpress site.

With the end of Google+, I want to try something new: I’ll host longer pieces on this blog, and post shorter messages on @juliank@mastodon.social. If you follow the Mastodon account, you will see toots for each new blog post as well, linking to the blog post.

Mastodon integration and privacy

Now comes the interesting part: If you reply to the toot, your reply will be shown on the blog itself. This works with a tiny bit of JavaScript that talks to a simple server-side script that finds toots from me mentioning the blog post, and then replies to that.

This protects your privacy, because mastodon.social does not see which blog post you are looking at, because it is contacted by the server, not by you. Rendering avatars requires loading images from mastodon.social’s file server, however - to improve your privacy, all avatars are loaded with referrerpolicy='no-referrer', so assuming your browser is half-way sane, it should not be telling mastodon.social which post you visited either. In fact, the entire domain also sets Referrer-Policy: no-referrer as an http header, so any link you follow will not have a referrer set.

The integration was originally written by @bjoern@mastodon.social – I have done some moderate improvements to adapt it to my theme, make it more reusable, and replace and extend the caching done in a JSON file with a Redis database.

Source code

This blog is free software; generated by the Hugo snap. All source code for it is available:

(Yes I am aware that hosting the repositories on GitHub is a bit ironic given the whole focus on privacy and self-hosting).

The theme makes use of Hugo pipes to minify and fingerprint JavaScript, and vendorizes all dependencies instead of embedding CDN links, to, again, protect your privacy.

Future work

I think I want to make the theme dark, to be more friendly to the eyes. I also might want to make the mastodon integration a bit more friendly to use. And I want to get rid of jQuery, it’s only used for a handful of calls in the Mastodon integration JavaScript.

If you have any other idea for improvements, feel free to join the conversation in the mastodon toot, send me an email, or open an issue at the github projects.

Closing thoughts

I think the end of Google+ will be an interesting time, requring a lot of people in the open source world to replace one of their main communication channels with a different approach.

Mastodon and Diaspora are both in the race, and I fear the community will split or everyone will have two accounts in the end. I personally think that Mastodon + syndicated blogs provide a good balance: You can quickly write short posts (up to 500 characters), and you can host long articles on your own and link to them.

I hope that one day diaspora* and mastodon federate together. If we end up with one federated network that would be the best outcome.

Ingo Juergensmann: Xen & Databases

14 October, 2018 - 02:21

I'm running PostgreSQL and MySQL on my server that both serve different databases to Wordpress, Drupal, Piwigo, Friendica, Mastodon, whatever...

In the past the databases where colocated in my mailserver VM whereas the webserver was running on a different VM. Somewhen I moved the databases from domU to dom0, maybe because I thought that the databases would be faster running on direct disk I/O in the dom0 environment, but can't remember the exact rasons anymore.

However, in the meantime the size of the databases grew and the number of the VMs did, too. MySQL and PostgreSQL are both configured/optimized to run with 16 GB of memory in dom0, but in the last months I experienced high disk I/O especially for MySQL and slow I/O performance in all the domU VMs because of that.

Currently iotop shows something like this:

Total DISK READ :     131.92 K/s | Total DISK WRITE :    1546.42 K/s
Actual DISK READ:     131.92 K/s | Actual DISK WRITE:       2.40 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 6424 be/4 mysql       0.00 B/s    0.00 B/s  0.00 % 60.90 % mysqld
18536 be/4 mysql      43.97 K/s   80.62 K/s  0.00 % 35.59 % mysqld
 6499 be/4 mysql       0.00 B/s   29.32 K/s  0.00 % 13.18 % mysqld
20117 be/4 mysql       0.00 B/s    3.66 K/s  0.00 % 12.30 % mysqld
 6482 be/4 mysql       0.00 B/s    0.00 B/s  0.00 % 10.04 % mysqld
 6495 be/4 mysql       0.00 B/s    3.66 K/s  0.00 % 10.02 % mysqld
20144 be/4 postgres    0.00 B/s   73.29 K/s  0.00 %  4.87 % postgres: hubzilla hubzi~
 2920 be/4 postgres    0.00 B/s 1209.28 K/s  0.00 %  3.52 % postgres: wal writer process
11759 be/4 mysql       0.00 B/s   25.65 K/s  0.00 %  0.83 % mysqld
18736 be/4 mysql       0.00 B/s   14.66 K/s  0.00 %  0.17 % mysqld
21768 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.02 % [kworker/1:0]
 2922 be/4 postgres    0.00 B/s   69.63 K/s  0.00 %  0.00 % postgres: stats collector process

MySQL data site is below configured max memory size for MySQL, so everything should more or less fit into memory. Yet, there is still a large amount of disk I/O by MySQL, much more than by PostgreSQL. Of course there is much I/O done by writes to the database.

However, I'm thinking of changing my setup again back to domU based database setup again, maybe one dedicated VM for both DBMS' or even two dedicated VMs for each of them? I'm not quite sure how Xen reacts to the current work load?

Back in the days when I did 3D computer graphic I did a lot of testing with different settings in regards of priorities and such. Basically one would think that giving the renderer more CPU time would speed of the rendering, but this turned out to be wrong: the higher the render tasks priority was, the slower the rendering got, because disk I/O (and other tasks that were necessary for the render task to work) got slowed down. When running the render task at lowest priority all the other necessary tasks could run on higher speed and return the CPU more quickly, which resulted in shorter render times.

So, maybe I experience something similar with the databases on dom0 here as well: dom0 is busy doing database work and this slows down all the other tasks (== domU VMs). When I would move databases back to domU this would enable dom0 again to better do its basic job of taking care of the domUs?

Of course, this is also a quite philosophical question, but what is the recommended setup? Is it better to separate the databases in two different VMs or just one? Or is running the databases on dom0 the best option?

I'm interested in your feedback, so please comment! :-)

Kategorie: DebianTags: DebianMySQLPostgreSQLXen 

Jeremy Bicha: Shutter removed from Debian & Ubuntu

14 October, 2018 - 01:29

This week, the popular screenshot app Shutter was removed from Debian Unstable & Ubuntu 18.10. (It had already been removed from Debian “Buster” 6 months ago and some of its “optional” dependencies had already been removed from Ubuntu 18.04 LTS).

Shutter will need to be ported to gtk3 before it can return to Debian. (Ideally, it would support Wayland desktops too but that’s not a blocker for inclusion in Debian.)

See the Debian bug for more discussion.

I am told that flameshot is a nice well-maintained screenshot app.

I believe Snap or Flatpak are great ways to make apps that use obsolete libraries available on modern distros that can no longer keep those libraries around. There isn’t a Snap or Flatpak version of Shutter yet, so hopefully someone interested in that will help create one.

Dirk Eddelbuettel: RcppNLoptExample 0.0.1: Use NLopt from C/C++

13 October, 2018 - 21:56

A new package of ours, RcppNLoptExample, arrived on CRAN yesterday after a somewhat longer-than-usual wait for new packages as CRAN seems really busy these days. As always, a big and very grateful Thank You! for all they do to keep this community humming.

So what does it do ?

NLopt is a very comprehensive library for nonlinear optimization. The nloptr package by Jelmer Ypma has long been providing an excellent R interface.

Starting with its 1.2.0 release, the nloptr package now exports several C symbols in a way that makes it accessible to other R packages without linking easing the installation on all operating systems.

The new package RcppNLoptExample illustrates this facility with an example drawn from the NLopt tutorial. See the (currently single) file src/nlopt.cpp.

How / Why ?

R uses C interfaces. These C interfaces can be exported between packages. So when the usual library(nloptr) (or an import via NAMESPACE) happens, we now also get a number of C functions registered.

And those are enough to run optimization from C++ as we simply rely on the C interface provided. Look careful at the example code: the objective function and the constraint functions are C functions, and the body of our example invokes C functions from NLopt. This just works. For either C code, or C++ (where we rely on Rcpp to marshal data back and forth with ease).

On the other hand, if we tried to use the NLopt C++ interface which brings with it someinterface code we would require linking to that code (which R cannot easily export across packages using its C interface). So C it is.

Status

The package is pretty basic but fully workable. Some more examples should get added, and a helper class or two for state would be nice. The (very short) NEWS entry follows:

Changes in version 0.0.1 (2018-10-01)
  • Initial basic package version with one example from NLopt tutorial

Code, issue tickets etc are at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Elana Hashman: PyGotham 2018 Talk Resources

13 October, 2018 - 11:00

At PyGotham in 2018, I gave a talk called "The Black Magic of Python Wheels". I based this talk on my two years of work on auditwheel and the manylinux platform, hoping to share some dark details of how the proverbial sausage is made.

It was a fun talk, and I enjoyed the opportunity to wear my Python Packaging Authority hat:

A very witchy @PyGotham talk from @ehashdn about dark ELF magic pic.twitter.com/W8JMEVW8GE

— Geoffrey Thomas, but spooky (@geofft) October 6, 2018 The Black Magic of Python Wheels Follow-up readings All the PEPs referenced in the talk

In increasing numeric order.

  • PEP 376 "Database of Installed Python Distributions"
  • PEP 426 "Metadata for Python Software Packages 2.0"
  • PEP 427 "The Wheel Binary Package Format 1.0"
  • PEP 513 "A Platform Tag for Portable Linux Built Distributions" (aka manylinux1)
  • PEP 571 "The manylinux2010 Platform Tag"
Image licensing info

Dirk Eddelbuettel: GitHub Streak: Round Five

13 October, 2018 - 08:11

Four years ago I referenced the Seinfeld Streak used in an earlier post of regular updates to to the Rcpp Gallery:

This is sometimes called Jerry Seinfeld’s secret to productivity: Just keep at it. Don’t break the streak.

and then showed the first chart of GitHub streaking

github activity october 2013 to october 2014

And three year ago a first follow-up appeared in this post:

github activity october 2014 to october 2015

And two years ago we had a followup

github activity october 2015 to october 2016

And last year we another one

github activity october 2016 to october 2017

As today is October 12, here is the newest one from 2017 to 2018:

github activity october 2017 to october 2018

Again, special thanks go to Alessandro Pezzè for the Chrome add-on GithubOriginalStreak.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Bastian Venthur: Introducing Litestats

13 October, 2018 - 05:00

Profiling in Python has always been easy, however, analyzing the profiler's output not so much. After the profile has been created you can use Python's pstats module but it feels quite clumsy and not really empowering. For Python 2 there has been RunSnakeRun, a very convenient tool for analyzing the profiler output, unfortunately that tool hasn't been updated since 2014. I recently ported it to Python3 and wxPython4 but I'm probably not going to maintain that code properly as I'm not very comfortable with wxPython.

I still wanted something nicer than pstats for profiling so I wrote litestats. Litestats is a simple command line tool that takes the output of the Python profiler and transforms the data into a sqlite3 database. You can now easily analyze the profiler output using sqlite on the command line, the sqlitebrowser for a graphical interface or use the data base as the foundation of your very own tooling around the analysis.

How does it work?

Litestats reads the dump of the profiler and creates a normalized data base with tree tables:

  • functions: contains each function (callers and callees) with filename, line number and function name
  • stats contains the statistics (primitive/total calls, total/cumulative time) for all functions
  • calls a caller-callee mapping

While this provides an exact representation of the dump, those tables would be cumbersome to use. So litestats additionally creates three views emulating pstats print_stats(), print_callers() and print_callees() functionality:

  • pstats
  • callers
  • callees
Installation

Litestats has no requirements other than Python itself and is available on PyPI:

$ pip install litestats
Usage
$ # run the profiler and dump the output
$ python3 -m cProfile -o example.prof example.py
$ # convert dump to sqlite3 db
$ litestats example.prof
$ # example.prof.sqlite created

You can now use the sqlite3 data base to investigate the profiler dump:

sqlite> select *
   ...> from pstats
   ...> order by cumtime desc
   ...> limit 20;

ncalls      tottime     ttpercall             cumtime     ctpercall   filename:lineno(function)
----------  ----------  --------------------  ----------  ----------  ------------------------------------
18/1        0.000161    8.94444444444444e-06  0.067797    0.067797    ~:0(<built-in method builtins.exec>)
1           1.0e-06     1.0e-06               0.067755    0.067755    <string>:1(<module>)
1           4.0e-06     4.0e-06               0.067754    0.067754    /usr/lib/python3.7/runpy.py:195(run_
1           6.0e-06     6.0e-06               0.066135    0.066135    /usr/lib/python3.7/runpy.py:62(_run_
1           1.1e-05     1.1e-05               0.066113    0.066113    /home/venthur/Documents/projects/lit
1           6.6e-05     6.6e-05               0.055152    0.055152    /home/venthur/Documents/projects/lit
1           4.1e-05     4.1e-05               0.0549      0.0549      /home/venthur/Documents/projects/lit
1           0.050196    0.050196              0.050196    0.050196    ~:0(<method 'executescript' of 'sqli
20/3        8.9e-05     4.45e-06              0.011064    0.003688    <frozen importlib._bootstrap>:978(_f
20/3        4.8e-05     2.4e-06               0.011005    0.00366833  <frozen importlib._bootstrap>:948(_f
20/3        7.5e-05     3.75e-06              0.01083     0.00361     <frozen importlib._bootstrap>:663(_l
15/3        3.5e-05     2.33333333333333e-06  0.01073     0.00357666  <frozen importlib._bootstrap_externa
29/5        2.5e-05     8.62068965517241e-07  0.010215    0.002043    <frozen importlib._bootstrap>:211(_c
3           6.0e-06     2.0e-06               0.010087    0.00336233  ~:0(<built-in method builtins.__impo
28/6        9.0e-06     3.21428571428571e-07  0.008977    0.00149616  <frozen importlib._bootstrap>:1009(_
1           9.0e-06     9.0e-06               0.00841     0.00841     /home/venthur/Documents/projects/lit
16          0.000138    8.625e-06             0.004802    0.00030012  <frozen importlib._bootstrap_externa
1           4.5e-05     4.5e-05               0.004143    0.004143    /usr/lib/python3.7/logging/__init__.
1           0.004038    0.004038              0.004038    0.004038    ~:0(<method 'commit' of 'sqlite3.Con
13          3.3e-05     2.53846153846154e-06  0.002368    0.00018215  <frozen importlib._bootstrap_externa

Dirk Eddelbuettel: binb 0.0.3: Now with Monash

12 October, 2018 - 19:36

The third release of the binb package just arrived on CRAN, and it comes with a new (and very crispy) theme: Monash. With that we are also thrilled to welcome Rob Hyndman as a co-author.

Here is a quick demo combining all (by now four) themes:

Also, Ista made the IQSS theme more robust to font selection. Other changes:

Changes in binb version 0.0.3 (2018-10-12)
  • The IQSS theme now has a fallback font if Libertinus is unavailable (Ista in #7)

  • Added support for 'Monash' theme (Rob Hyndman in #10 and #11 closing #9)

  • Simplified some options for the 'Monash' theme (Dirk in #13)

  • The IQSS theme can now set an alternate titlegraphic (Ista in #14)

CRANberries provides the usual summary of changes to the previous version.

For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Deepanshu Gajbhiye: Google Summer of code at Debian Final Report

11 October, 2018 - 23:26
Google Summer of code 2018Table of contents
  1. Introduction
  2. Final Summary
  3. Deliverable
  4. Weekly reports & Blog posts
  5. Other contributions
  6. Thank you
Introduction

Virtual LTSP server project automates installation and configuration of LTSP server with vagrant. It is the easiest way to create LTSP setup. We have developed the project to do the same for Linux mint 19 and Debian 9. We also created several scripts for testing, create ltsp client, manage accounts, etc. Also created packer scripts to create vagrant boxes that we will use in the project.

Final Summary

Google Summer of Code was a great opportunity to work with really smart and amazing people. I learned a lot in the process. It took my understanding of Vagrant, bash-scripting, packer, ltsp, Debian packaging to a whole another level. I started with a basic provisioner script to install ltsp to vagrant box. Then we make several improvements in it.

Later made several features and improvements in it. After that a major issue was the client was unable to boot. In order to solve this issue I searched through all the content on the internet about ltsp. Even asked the active developers of ltsp on how to fix this issue. They have been working on ltsp since 2006 but they haven’t encountered this problem yet. After struggling for lots of I solved it! Looking at the complexity and the time it took mentors told me to write a separate blog post about it. We have also created Virtual ltsp server for Debian 9. Also one for linux mint 19.

I had to create a new linux mint vagrant box with xfce. Which was really fun. I also automated its creation with packer scripts. We also did port from Edubuntu packages to Debian. It is locally built and installed via a small script. In the end, we added features like automatic login, guest login, and several scripts to optimize the workflow for the user. This was a really short summary of the work done. More details can be found on the weekly reports.

DeliverablesVirtual LTSP Server

docker-scripts/dev--LTSP

Pull request made

docker-scripts/dev--LTSP

Commits made
  • bionic branch —
  • buster branch —
Issues worked on

docker-scripts/dev--LTSP

Packer scripts to create vagrant box

docker-scripts/dev--LTSP

Linux mint tara vagrant box

Vagrant Cloud by HashiCorp

Ported edubuntu packages from ubuntu to debianDocumentation

docker-scripts/dev--LTSP

Weekly reports & Blog posts
  • Week1:

GSoC weekly report of Deepanshu Gajbhiye week 1

  • Week2:
  • Week3:
  • Week4:
  • Week5:
  • Week6:
  • Week7:
  • Week8:
  • Week9:
  • Week10:
  • Week11:
  • Week12:
Other contributions

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=905006

Thank you

I am very thankful to Google and Debian for accepting me to Google Summer of Code. Working for GSoC was an amazing experience. I will definately participate next year as well.

Thank to my mentors Dashamir Hoxha and Akash Shende for their solid support, quick response, patience and trust on me.

Thank you Daniel Pocock for encouragement and Big thanks to Debian, ltsp, Vagrant community for being very helpful.

Antoine Beaupré: Archived a part of my CD collection

11 October, 2018 - 22:52

After about three days of work, I've finished archiving a part of my old CD collection. There were about 200 CDs in a cardboard box that were gathering dust. After reading Jonathan Dowland's post about CD archival, I got (rightly) worried it would be damaged beyond rescue so I sat down and did some research on the rescue mechanisms. My notes are in rescue and I hope to turn this into a more agreeable LWN article eventually.

I post this here so I can put a note in the box with a permanent URL for future reference as well.

Remaining work

All the archives created were dumped in the ~/archive or ~/mp3 directories on ?curie. Data needs to be deduplicated, replicated, and archived somewhere more logical.

Inventory

I have a bunch of piles:

  • a spindle of disks that consists mostly of TV episodes, movies, distro and Windows images/ghosts. not imported.
  • a pile of tapes and Zip drives. not imported.
  • about fourty backup disks. not imported.
  • about five "books" disks of various sorts. ISOs generated. partly integrated in my collection, others failed to import or were in formats that were considered non-recoverable
  • a bunch of orange seeds piles
    • Burn Your TV masters and copies
    • apparently live and unique samples - mostly imported in mp3
    • really old stuff with tons of dupes - partly sorted through, in jams4, reste still in the pile
  • a pile of unidentified disks

All disks were eventually identified as trash, blanks, perfect, finished, defective, or not processed. A special needs attention stack was the "to do" pile, and would get sorted through the other piles. each pile was labeled with a sticky note and taped together summarily.

A post-it pointing to the blog post was included in the box, along with a printed version of the blog post summarizing a snapshot of this inventory.

Here is a summary of what's in the box.

Type Count Note trash 13 non-recoverable. not detected by the Linux kernel at all and no further attempt has been made to recover them. blanks 3 never written to, still usable perfect 28 successfully archived, without errors finished 4 almost perfect: but mixed-mode or multi-session defective 21 found to have errors but not considered important enough to re-process total 69 not processed ~100 visual estimate

Mario Lang: "Learning from machines" by Ashi Krishnan

11 October, 2018 - 16:00

I need to share this talk by Ashi Krishnan because I think it is very well done, and I find the content extremely fascinating and interesting.

If you are into consciousness and/or dreaming, do yourself a favour and allocate 40 minutes of your life for this one.

Thanks Ashi, you made my day!

Pages

Creative Commons License ลิขสิทธิ์ของบทความเป็นของเจ้าของบทความแต่ละชิ้น
ผลงานนี้ ใช้สัญญาอนุญาตของครีเอทีฟคอมมอนส์แบบ แสดงที่มา-อนุญาตแบบเดียวกัน 3.0 ที่ยังไม่ได้ปรับแก้