You are here

Web Services

Topics:


Web Services in General:
In General, Minerva web services is a standard Linux-Based web hosting services. Services are provided through Apache 2.4 with the MPM-ITK addon. It is designed to support basic web pages for the intent and purpose of providing web-based science portals and other applications to interact with the jobs, queues, and large science datasets on Minerva. The web services should be on-par with common unix-based web hosting services such as Dreamhost etc. Most common web frameworks and applications should easily fit into this service such as: Django, Wordpress, Wikimedia, Custom Perl apps, and static content.

Server-side content is not run as the common "apache" user, but instead as the owner of that application directory. That is, each user's content/scripts run as that user.

Each user's default web services landing point is <username>.u.hpc.mssm.edu

Account Creation Time Delay: It may take between 30min - 1hour after user account creation for the system to automatically subscribe a user to web services. If you are a new user, please wait the delay and check web services again before submitting a trouble ticket.


Usage, Security / SSL, Availability and Accessibility:
The Minerva web services are designed to support basic web pages for the intent and purpose of providing web-based science portals and other application interactions and to interact with the jobs, queues, and large science datasets on Minerva. The web services should be on-par with common unix-based web hosting services such as Dreamhost etc. That said, we don't recommend using web services as a primary website for a department or lab, websites with heavy traffic, or for websites that require a guaranteed amount of uptime. There are no uptime guarantees. Web services may go down during Preventative Maintenance periods. We may block IPs with suspicious activity for periods of time, possibility indefinitely, possibly automatically.
One possible usage style may be to create a website for your department or lab elsewhere and pull specific real-time generated content from the Minerva web services via javascript includes, redirects, and API's.

Usage is unlimited for the purposes of interaction with Minerva components and viewing science.

At this time, web services are only accessibly from on-campus at Mt Sinai. Pending some DNS changes and testing, web services should be accessible world-wide.

WARNING WARNING WARNING: Be careful! Content, executables, scripts, symlinks, applications, etc within the www/ folder may be ( or are ) world accessible. Scripts and applications launched via Apache in that folder run as your user! They can access any data (including your groups' /project data), delete data, archive data, submit jobs, cancel jobs, email people, etc as your user. You are responsible for any actions taken on your behalf!

SSL
The initial deployment of Web Services does not have SSL enabled.
SSL is planned and in the process of being implemented. Once implemented, it will be required for authentication via passwords.


Getting Started with User-based Web Services:
Each user, by default, has a web services site. A specific user's site is located at <username>.u.hpc.mssm.edu The Document Root for a user's site is within their home folder in a folder called www.
Thus, <username>.u.hpc.mssm.edu points to ~username/www

Step 1:
If this folder does not exist in your home directory, you should create it.
$ mkdir ~/www

Step 2:
Place content in the www folder.

$ cat > ~/www/index.html <<EOF
Hello World from my website.
EOF

Step 3:
Test your web services. Point your browser to http://yourusername.u.hpc.mssm.edu


Supported Scripting Languages:
In Short:
The following languages are supported: PHP, Python, Perl, and CGI (all others).
The Default extensions for each language:

Language Extension Index
PHP .php index.php
Python (mod_python) .py index.py
Python (wsgi) See Below See Below
Perl .pl index.pl
CGI .cgi index.cgi


PHP:
Web Services supports PHP. Files must end in a .php extension. The service is based on PHP 5.3.3. The following modules / patches are installed: gd ldap mysql pear xml pear-DB pgsql mbstring pecl-imagick imap snmp suhosin.
Clean URL's may be supported through a .htaccess modification. Please read the PHP and your application's documentation for how it should be done for your application.

An example Hello World in PHP:

<?php
print "Hello World";
?>


Python:

mod_python method:
We support server-side python execution through mod_python. Files ending in a .py extension are interpreted as python.
To make applications such as Django and other python application function, one should follow the instructions for mod_python, which may mean making an "connector file" that calls the application through a function call. Often a .htaccess will be required to override the directory handler to make Clean URL's work. Functions within a python file may be called via modifications to .htaccess. See some examples here: Apache Configuration Web Python
The service is based on Python 2.6.6, the system python. If one of the /packages pythons is desired, please submit a ticket describing the interest.

A "Hello World" in mod_python:

def index():
   s = """\
<html>
<body>
<h2>Hello World!>
</body>
</html>
"""
   return s


wsgi method:
The WSGI method of hosting python programs is new and more elegant compared to the mod_python method. It is now the standard for many frameworks, such as Django, Pyramids, etc.
( Please note this is WSGI-Script, not the daemonized / threaded style. Most apps should be able to handle this correctly. )
The Key to hosting a WSGI app is creating a directory in which you put the WSGI file, which will serve as the root of the app.
For example, if you want to host a app at https://YOURUSER.u.hpc.mssm.edu/myapp/ such that the URLs https://YOURUSER.u.hpc.mssm.edu/myapp/subpage and https://YOURUSER.u.hpc.mssm.edu/myapp/subpage/anotherthing are all handled by your WSGI app, then, do the following:
Create the folder by your main app name inside the root of your www folder:

mkdir ~/www/myapp

Then add a .htaccess file to declare that all files in that folder will be WSGI files. Put the following lines in a .htaccess file, like so:
Example of how to edit:

vi ~/www/myapp/.htaccess

Line to put in the file:

SetHandler wsgi-script

At this point, if you create a test WSGI app in your folder it will work, for example:

$ cat wsgione.wsgi
def application(environ, start_response):
    status = '200 OK'
    output = 'Hello World!'

    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]

    start_response(status, response_headers)

    return [output]
$

This app above would be available at https://YOURUSER.u.hpc.mssm.edu/myapp/wsgione.wsgi URLs beyond that would also be handled by it, for example: https://YOURUSER.u.hpc.mssm.edu/myapp/wsgione.wsgi/one/two/three
(In this case, of course, nothing is done with those URLs, but something could be in your app, like with django).
Of course, this URL is not very clean, so we can move the file from wsgione.wsgi to wsgione ( just removing the .wsgi ) and it will still work. Thus, you could host many apps in one folder, for example, /apps/app1, /apps/app2 .. etc etc.

That too, still may not be ideal for some use cases, so finally, it is possible to make a main top-level folder redirect into an app, however, ideally you should make a folder that only contains the one WSGI file and the .htaccess file.
You will need to call your WSGI file index.wsgi.
Then set your .htaccess file like so:

AddHandler wsgi-script .wsgi
DirectoryIndex index.wsgi
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ index.wsgi/$1 [QSA,PT,L]

This will rewrite https://YOURNAME.u.hpc.mssm.edu/myapp/one/two/threee into the index.wsgi file and pass-in the URL parameters for the app to handle.

Perl and CGI:
Perl is supported through mod_perl and CGI. Simply end your file with a .pl or .cgi extension and ensure it is executable on the filesystem. (for example do a chmod +x test.pl to make it executable). As the Perl is being executed as CGI, you must put a "shebang" at the beginning of your file to point to the perl executable. The service is based on Perl 5.10.1
The following Perl modules are installed for sure, in addition to possibly others: perl-DBI perl-URI perl-IO-Socket-SSL perl-Net-DNS perl-HTML-Tagset perl-Archive-Tar perl-Net-IP perl-DBD-MySQL perl-Socket6 perl-Digest-SHA1 perl-IO-Socket-INET6 perl-Compress-Zlib perl-DBD-Pg perl-HTML-Parser perl-IO-Zlib perl-BSD-Resource perl-SGMLSpm perl-libwww-perl perl-Digest-HMAC perl-Net-SSLeay perl-Pod-POM perl-Pod-Simple perl-Date-Calc

An example Perl Hello World:

#!/usr/bin/perl
print "Content-type: text/html\n\n";
print "Hello, world!\n";

As Perl here is executed as if it was CGI, you may instead create any executable file, including shell script (bash, csh, ksh) or even an executable binary with an appropriate name to execute it.


Authentication:
It may be desirable to force visitors to authenticate before viewing a portion (or all) of your website.
Authentication is supported in two ways: System Auth or Password File

System Auth
System Auth will authenticate users using their Mt Sinai / Minerva Credentials. For the campus-facing web servers, password auth is supported. For public-facing web servers, two-factor auth is required. You must use the appropriate auth for the server you are using.
To enable System Auth, you should create a .htaccess file in the root of the directory structure you wish to protect.

The .htaccess file to use for System Auth is:

AuthType Basic
AuthName Your-Site-Name
AuthBasicProvider external
AuthExternal pwauth
require valid-user

Password File
Password File will allow you to create a file with usernames and passwords combinations that will allow successful logins. Password File is not secure. It allows you to create usernames and passwords such as user "test" with password "test". It's not for serious authentication, however, it can be useful for light protection of folders that you just don't want Google to accidentally run in to, for example. Please do not create bogus passwords such as test or password
To enable Password File auth, you will need to create a password file and create a .htaccess file in the root of the directory structure you wish to protect.

The password file should sit somewhere that is outside of the www/ folder. A good place may be in a folder in your home directory. If you desire, you can create a folder in your home directory for just storing these types of file, or possible a hidden folder in your home directory (one beginning with a dot)

To create a folder and password file and user named test:

$ mkdir ~/.htpasswords
$ htpasswd -c ~/.htpasswords/passdb1 test
New password:
Re-type new password:
Adding password for user test

The .htaccess file to use for Password File is:

AuthType basic
AuthName "private area"
AuthUserFile    "/hpc/users/user1/.htpasswords/passdb1"
Require            valid-user
Order allow,deny
Allow from all

Note: You should place the full and proper path to the password file in the AuthUserFile line.

General:
You may prefer some variations. For example:
Varing the line AuthName will show the user a different prompt at the login box.
Varing the require valid-user to require user user1 user2 will restrict access to user1 and user2 only. Similarly, require group group2 will restrict access to group2
Other variations are possible. Anything that may be supported in a .htaccess file is allowed (some restrictions). You can block based on remote IP etc.
Some Sites show many possible variations.

Please read the Apache documentation on Basic Auth and The Require Directive


Group-based Web Services:
As described above, web services are currently only created, by default, for each user. For a group-based web service, or a custom-named web service, please submit a ticket by emailing hpchelp@hpc.mssm.edu with your request. Do note that the web service will need to run as some specific user, so please plan ahead as to which user should own the web service.
Similarly, custom web services (see below) on a per-user or per-group basis may use non-default domain names, however, it is a special request.


Domain Names:
By default, the Scientific Computing group places all services below the hpc.mssm.edu domain. For automated purposes, we place the generated web service accounts below the u.hpc.mssm.edu domain (u for "users"). Although this provides a quick way for users to start working with web services, it may not be desirable a desirable URL for certain cases. Shorter domains typically look nicer. Long URL's with usernames can be confusing.

The Scientific Computing group does not have the ability to acquire or control URL's outside of our own domain. However, your user / group may go through either Mt Sinai IT to acquire some other *.mssm.edu (or similar) domain and/or purchase a domain from a public entity, such as Network Solutions, ENOM, or Godaddy. You will also need some sort of Domain Name Hosting services. You can then create an A or CNAME record to point to our web hosting IP address(es). We can configure your account to accept requests for these additional domains.

The Scientific Computing group and the web services do not provide Domain Name Hosting or Email Services.

SSL
SSL with domain names outside the hpc.mssm.edu domain presents some challenges. Each unique domain will require an additional IP Address. This will need to be evaluated on a case-by-case basis to see if there are remaining addresses to address the need. Please plan ahead before making any purchases to ensure the service will function correctly.
Other hybrid variations may provide useful. For example, the non-secure content can be on the custom domain and the secure content can be on the default u.hpc.mssm.edu domain. The two sites can pass the user back and forth as needed.


Custom Server:
The Scientific Computing group does have the ability to create custom servers for applications which require deep levels of specific versions of packages or other custom software, such as Apache Tomcat. Server availability is limited and evaluated on a case-by-case basis. Please submit a ticket with your request by emailing hpchelp@mssm.edu

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer