CoCalc Blog

Project Initialization Scripts

Harald Schilly • • cocalc

Under the hood, CoCalc provides you with a very flexible and versatile Linux environment. Due to the upgrade to Kubernetes, our (unofficial) support for using crontab files for periodic tasks was removed. Instead, there is a new and more flexible mechanism to use regular Bash, Python, or SageMath files to accomplish even more.

Project Initialization

When a CoCalc project starts up, an instance of Supervisor is started and responsible for running:

Initialization file management

Supervisor is configured to run this initialization file like background processes. This means it checks the exit value of the process and restarts it if this code is not 0 or 2. Therefore, a script that runs without an error is not restarted.

On the other hand, if the process is terminated, interrupted or crashes – therefore the exit code is e.g. 1 – the process is restarted. This adds some robustness to long running background jobs.

Example 1: record start time

A very simple example is to record the project’s start time. Go to your home directory and create a file project_init.sh with that content:

date > project-start

This is a very simple bash script, which pipes the output of the date command into the file project-start.

In order to see its effect, the file needs to be saved and the project restarted. Give it a few seconds to come back online and run the script. After that (maybe click the refresh button in the file listing) you should see this file and its content might look like:

Mon Sep 12 11:14:20 UTC 2017

Other languages besides Bash?

You can run any language via bash’s exec! For example, project_init.sh containing

exec python3 project_init.py

will run a Python 3 initialization file named project_init.py.

Using SSH with CoCalc

Hal Snyder • • cocalc

CoCalc support for SSH is significantly improved with the CoCalc upgrade to Kubernetes. This article reviews new features and gives examples using SSH with CoCalc.

New SSH Features

TL;DR: SSH to CoCalc projects has been completely rewritten. For example, you can now paste a single ssh key in account settings, and it will automatically be available in all of your projects!

User Interface for Managing Public Keys

User interfaces are now available for managing your SSH keys. Using these forms, you can add and remove keys, as well as viewing which keys are configured and when they have been used last. There are separate UI dialogues for keys configured at account and project scope, as described below.

To connect from a remote computer to the SSH server on a project, you must have owner or collaborator status on that project. It is NOT required that the project have network access enabled; network access is only required if you want to ssh from a project to somewhere else.

NOTE: Manual editing of the authorized_keys file is no longer supported. Use the procedures below to add public keys.

Configure a single key for all projects for a given account

It is now possible to set a public key for your CoCalc account that works with all projects where your account is owner or collaborator. Click on your account name or the gear icon next to it, and choose the SSH Keys tab. There is a form at right for entering keys, as shown here:

To add a key, enter a descriptive title and paste the .pub file for the public key file into the form and click Add SSH Key. Extraneous spaces and newlines in the key that is pasted in will be automatically removed.

The title, MD5 fingerprint, and last-used timestamp will be displayed for each key that is added, as shown here:

Multiple keys can be added in this way.

If you want to remove an account key, return to the SSH Keys tab and click the Delete button to the right of the key you want to delete.

To use an account key, you will need the ssh user and hostname for a project you want to access. Open project settings (wrench icon) and look in the SSH Keys section at lower left for the information under ‘Use the following username@host:’. You will see that the username is the same as the destination project id without hyphens, and the host is always ssh.cocalc.com. (This is a major improvement over before when the host depended on the project.)

See below for detailed examples of using SSH keys.

Configure a public key for a single project

There is also a form to add and remove keys for a single project. To add a key for use with a single project, open project Settings (wrench icon) and scroll down to the SSH Keys section at lower left. Click ‘Add an SSH Key’ to open the dialogue shown below.

SSH agent forwarding

If you’re familiar with ssh-agent forwarding, this feature is now supported by CoCalc. You will need to have ssh-agent configured on your local system and have public keys distributed to all target hosts for agent forwarding to work.

See Steve Friedl’s Tech Tips guide for more info about SSH agent forwarding.

SSH access between CoCalc projects

When the client and server for an SSH connection are both CoCalc projects, you can use a hostname of ssh rather than ssh.cocalc.com. Using the shorter hostname gives a much faster connection, because it is optimized within the CoCalc cloud.

SageMathCloud is Now CoCalc

Hal Snyder • • cocalc

As of May 20, 2017, SageMathCloud is being renamed to CoCalc, for Collaborative Calculation in the Cloud.

A Natural Evolution

SageMathCloud was introduced in April 2013 as a hosted platform for the SageMath software system. It saved thousands of users the time and effort of building, installing, and updating SageMath locally. SageMathCloud provided a single consistent platform for improving the range and quality of services offered.

Over time, we began adding new features, mostly for our own needs: a course manager to teach students SageMath, chat to coordinate within the platform, a $\LaTeX$ editor to write documents, and Jupyter notebooks to support fellow instructors. In addition to conventional backups, we added snapshots and time travel. We added support for numerous languages, including Python 3 as well as Python 2, R (including R markdown), the complete Anaconda distribution, Octave, Julia, C, C++, Haskell, and Scala. We included major libraries and packages for these languages, and update them regularly.

We became a platform that enables students, teachers, and researchers to collaborate using a broad selection of programming languages and development tools – not just SageMath.

While SageMathCloud suggests exactly what we started out as, we have certainly outgrown it. The new name, CoCalc, reflects the evolution into a general purpose, cloud-based calculation and computation platform. Our recent move to rewrite the Jupyter notebook to be more integrated and fully supported is an example of how our mentality has shifted.

The CoCalc Experience

Newcomers can explore CoCalc services from different approaches:

What Should I Do If I’m Already an SMC Subscriber?

When you visit https://cloud.sagemath.com after the change, you’ll be immediately redirected to https://cocalc.com. You’ll then have to sign in again the first time. That will set a cookie, and after that everything will be as normal. Anyone following a link into the old name, e.g. https://cloud.sagemath.com/some_path..., will be redirected to the same path under the new name, https://cocalc.com/some_path....

You will notice our documentation changing to use the CoCalc product name and the CoCalc.com domain name.

We have created a new CoCalc discussion group in Google Groups here and we will automatically subscribe all current sage-cloud subscribers to it.

What Stays the Same

The company behind CoCalc will remain SageMath, Inc. Our staff email addresses will remain in the sagemath.com domain.

More importantly, we remain committed as ever to improving the experience of scientific and exploratory computing for students, teachers, and researchers around the world.

Try CoCalc Today

If you’re not already a subscriber and wonder what all the fuss is about, now is a great time to get acquainted. Start CoCalc for free today.

New SMC Jupyter Client Released

Hal Snyder • • jupyter

Today SageMath, Inc. is pleased to announce the release of a completely new Jupyter notebook implementation, optimized for SageMathCloud, our cloud-based, collaborative environment.

This new implementation was motivated by the need for better collaboration and browser/server synchronization. Rewriting also offered an opportunity to integrate previously-unavailable features of SMC into Jupyter notebooks. However, throughout this process we minimized interface discrepancies between old and new versions so that switching is as seamless as possible.

Here’s an overview of the most significant enhancements in today’s release:

Collaboration

Simultaneous Editors

Simultaneous editing by multiple people is now fully supported, including multiple cursors and document-wide, user-specific undo and redo.


User One and User Two editing a notebook at the same time.

Chat

Each notebook now has a chat sidebar. As always, chat supports markdown, $\LaTeX$, and video chat.


Text chat with markdown shown in panel to right of notebook body.

Server-side enhancements

TimeTravel

Jupyter includes TimeTravel just like our other editors. It shows detailed history of all changes to a notebook, and the author of each change.


Time Travel view of previous notebook as seen by User Two
showing revision 93 (of 95) created by User One.

Output

User interface


Sample notebook about to be exported.



Exporting notebook to Latex and opening the .tex file in SMC.



Converted ipynb file opened in LaTeX editor.

RethinkDB versus PostgreSQL: my personal experience

William Stein (wstein@sagemath.com) •

Introduction

Initially motivated by the shutdown of the RethinkDB company, and the licensing situation with RethinkDB (a blocker for certain parts of my business), I worked very hard for two months to completely rewrite the realtime and database components of SageMathCloud (SMC) to use PostgreSQL instead of RethinkDB, initially motivated by this discussion in Hacker news. I battled with and used RethinkDB heavily since May 2015, and I’ve used PostgreSQL heavily as well, with production data, rewriting all the same queries in both systems, so I’m in a good position to compare them for my use case (the site SMC).

This is my story. It’s a personal comparison, with NO BENCHMARKS or hard data you could reproduce. It’s what I would tell you if we were talking by the water cooler.

Summary:

SMC for Collaborative LaTeX Editing

Hal Snyder • • latex

SageMathCloud (SMC) is the most powerful online $\LaTeX$ collaboration software available today. SMC offers the full complement of features expected from online services today for professionals and students, on a par with and exceeding other leading products, such as Overleaf and ShareLaTeX. In addition, SMC offers a complete environment for teaching, research, and exploratory computing with the same rich feature set.


Here’s an overview of key SMC features to back up the claims of the previous paragraph.

Inline LaTeX Errors

haraldschilly • • latex

Small update of our LaTeX editor. Now it shows LaTeX errors inline, which should make it easier to fix these problems. The parser reading the LaTeX log was also improved to treat consecutive errors separately. Error location mapping also works for .Rnw knitr source files.

Wishbone

haraldschilly • • r

Do you want to run Wishbone in your SMC project?

Wishbone is an algorithm to align single cells from differentiation systems with bifurcating branches. Wishbone has been designed to work with multidimensional single cell data from diverse technologies such as Mass cytometry and single cell RNA-seq.

First, open a terminal file and run the following lines to install it locally inside your project:

pip3 install --user Cython pip3 install --user git+https://github.com/dominiek/python-bhtsne.git pip3 install --user git+https://github.com/jacoblevine/phenograph.git pip3 install --user git+https://github.com/ManuSetty/wishbone.git

Then, open a new Jupyter notebook, switch to the “Python 3” kernel, and run examples from their documentation.

(In order to access GitHub or PyPi, your project needs “internet access”)

R updated

haraldschilly • • r

Dear R users!

We’ve switched our default R to be the “official” one from https://www.r-project.org/. Therefore the default R version has been updated from version 3.2.4 to the most recent one 3.3.2.

This is a quite significant update from the one in SageMath, hence this notice. If for some reason you still need to work with the older version of R in SageMath, do this:

  1. In the command-line use R-sage instead of R and Rscript-sage instead of Rscript.

  2. In Jupyter notebooks there are two R kernels: the newer “R (R-Project)” and “R (SageMath)” for the version shipped with SageMath.

Regarding libraries, we installed all the ones we know about and many more. In total, there are 557 R libraries (1.3GB) available. Is something that you need still missing? Email help@sagemath.com.

using SMC with python & data science MOOC

halsnyder • • python, mooc, and datascience

Here are a couple tips based on my experience using SMC to complete the Coursera course, Introduction to Data Science in Python. This course is the first installment of a new 5-part Applied Data Science with Python Specialization from the University of Michigan.

Examples and study assignments for the course are offered as Python3 Jupyter notebooks, i.e. .ipynb files. Students may use Coursera-hosted jupyter notebooks or any other platform that allows them to run the code. Homework is submitted by uploading a .ipynb file for each programming assignment.

The following might be helpful for students taking the course and using SMC:

1. Set Jupyter kernel.

After uploading a course .ipynb file, change the kernel from Python3 to Anaconda (Python3) as shown below. This will prevent errors such as AttributeError: ‘Index’ object has no attribute ‘str’ due to different versions of pandas.

2. Convert Jupyter notebooks to Sage worksheets.

If you would rather code in a Sage worksheet than a Jupyter notebook, use the SMC script smc-ipynb2sagews to convert the files. Open a terminal file, for example mooc.term, and enter the following commands

$ cp Assignment\ 2.ipynb assgn2.ipynb
$ smc-ipynb2sagews assgn2.ipynb
/usr/local/bin/smc-ipynb2sagews: Creating SageMathCloud worksheet 'assgn2.sagews'
$ open assgn2.sagews