How to Make a Smartphone Controlled 3D Web Game

The following is a guest post by Charlie Walter. Charlie does a bunch of work with Three.js (3D in the browser with WebGL) and gaming concepts. If that stuff interests you, read on!

In this tutorial I’ll explain an approach on how to connect your smartphone to a 3D web game. We’ll be making a car which you can control by tilting your phone (using the phone’s accelerometer). We’ll use the JavaScript library three.js for working with WebGL as well as WebSockets through the socket.io library and a few other web technologies.

Try it now

Here’s the live demo you can play with right now. Note that it works best on WiFi.

Getting Set Up

You are going to need to install Node if you haven’t already. We are going to be using Express to set up our server, and socket.io for the WebSocket communication.

Create a directory for this project, and put this package.json at the root:

{
  "name": "smartphone-controller-game",
  "version": "0.1.0",
  "devDependencies": {
    "express": "*"
  }
}

Now open up your project directory in terminal and install your project dependencies with this command:

npm install

This looks in the package.json file and uses the devDependencies object to install the correct dependencies and versions of those dependencies. NPM uses Semvar versioning notation, and “*” means “latest”.

In order to use socket.io we need to set up a server. This can be done using Express. First, let’s serve up an index file. Create a file that will contain our server code. Call it server.js:

var express = require('express'),
    http    = require('http'),
    app     = express(),
    server  = http.createServer(app),
    port    = 8080;

server.listen(port);

app

  // Set up index
  .get('/', function(req, res) {

    res.sendFile(__dirname + '/index.html');

  });

// Log that the servers running
console.log("Server running on port: " + port);

This sets up a server running on port :8080. When the root is requested (“/”) it will send the `index.html` file in the response.

Create the index.html file:

<!DOCTYPE html>
<html lang="en">

<head>
  <title>Smartphone Controller Game</title>
</head>

<body>
  Hello World!
</body>

</html>

Now run this in the terminal:

node server.js

It should say:

Server running on port: 8080

Open the URL localhost:8080 in a browser and your index file should be rendered!

Getting Sockets Going

Now that we have our project set up, let’s get our client and server communicating via sockets. First we need to install socket.io.

npm install socket.io --save

In index.html, include socket.io in the <head>:

<script src="/socket.io/socket.io.js"></script>

and after the opening <body> tag add:

<script>
var io = io.connect();

io.on('connect', function() {
  alert("Connected!");
});
</script>

This connects to socket.io and alerts a message letting us know it’s working.

In server.js add this:

var io = require('socket.io').listen(server);

io.sockets.on('connection', function (socket) {
  console.log("Client connected!")
});

This sets up socket.io using the server and logs when new clients connect.

Since server code has changed, we will have to re-run the server. Press “Ctrl + C” in the terminal to cancel the current process. This will have to be done each time server.js is updated.

We should now see the alert notifying us of a successful socket.io connection and the terminal should log “Client connected!” almost instantly.

Connecting the Phone

Now we will connect a browser window on a phone (the controller for the car) to a desktop browser (the game). Here’s how that will work:

The game client will tell the server it wants to connect as a game
The server will then store that game socket and then tell the game client it’s connected
The game client will then create a URL using its socket ID as a URL parameter
The phone (or any other tab/window) will then go to this link and tell the server it wants to connect as a controller to the game socket with the ID in its URL
The server will then store that controller socket along with the ID of the game socket it’s connecting to
The server then assigns that controller socket’s ID to the relevant game socket object
The server then tells that game socket it has a controller connected to it and tells the controller socket it has connected
The relevant game socket and controller socket will then alert()

Let’s get the game client telling the server it’s connecting as a game client. In place of the alert(), add:

io.emit('game_connect');

This emits an event we’ve named game_connect. The server can then listen for this event and store the socket and send a message back to the client to tell it that it’s connected. So, add this as a new global variable:

var game_sockets = {};

Then add this in place of the console.log():

socket.on('game_connect', function(){

  console.log("Game connected");

  game_sockets[socket.id] = {
    socket: socket,
    controller_id: undefined
  };

  socket.emit("game_connected");
});

The controller_id will be populated with the socket ID of the controller connected to this game, when it connects.

Now restart the server and refresh the client. The terminal should now be logging the game connections.

Game connected

Now that the server emits an event called game_connected specifically to that socket (which emitted the game_connect), the client can then listen for this and create the URL:

var game_connected = function() {
  var url = "http://x.x.x.x:8080?id=" + io.id;
  document.body.innerHTML += url;
  io.removeListener('game_connected', game_connected);
};

io.on('game_connected', game_connected);

Replace x.x.x.x with your actual IP. To get this, you can use `ifconfig` from your Mac/Linux terminal or `ipconfig` in the Windows command prompt. This is the IPv4 address.

When you restart the server and go to the client, a URL for your IP on port :8080 with an ID parameter on the end should be present.

Great! When that URL is copied into another tab (or manually entered into a phone) nothing else happens other than creating another URL. That’s not what we want. We want it so that when this URL (with the ID parameter) is navigated to, the client should recognize that it has this parameter and tell the server to connect as a controller instead.

So, wrap everything inside of the io.on('connect', function() { in the else of this:

if (window.location.href.indexOf('?id=') > 0) {

  alert("Hey, you're a controller trying to connect to: " + window.location.href.split('?id=')[1]);

} else {

  // In here

}

Load up the client and when you navigate to a created URL, it will alert you that you are trying to connect as a controller, rather than a game. Here we will connect it to the server and link it up with the relevant game socket.

Replace the alert() with:

io.emit('controller_connect', window.location.href.split('?id=')[1]);

This emits an event called controller_connect and sends the ID, which we have in the URL, to the server. Now the server can listen for this event, store the controller socket, and connect it to the relevant game. First we need a global variable to store the controller sockets:

var controller_sockets = {};

Inside io.sockets.on('connection', function (socket) { } add this:

socket.on('controller_connect', function(game_socket_id){

  if (game_sockets[game_socket_id] && !game_sockets[game_socket_id].controller_id) {

    console.log("Controller connected");

    controller_sockets[socket.id] = {
      socket: socket,
      game_id: game_socket_id
    };

    game_sockets[game_socket_id].controller_id = socket.id;

    game_sockets[game_socket_id].socket.emit("controller_connected", true);

    socket.emit("controller_connected", true);

  } else {

    console.log("Controller attempted to connect but failed");

    socket.emit("controller_connected", false);
  }

});

This checks whether there is a game with that ID and confirms that it doesn’t have a controller connected to it already. The server emits an event on the controller socket called controller_connected and is sent a boolean depending on this. Its success is also console.log()‘d. If the check is successful then the new controller socket is stored along with the ID of the game it is connecting to. The controller socket ID is also set on the relevant existing game socket item.

Now the terminal should show when a controller is connected to the game. If we attempt to connect a second controller, it will fail (due to the second part of the validation).

Server running on port: 8080
Game connected
Controller connected
Controller attempted to connect but failed

Also, if we edit the URL and try to connect to a random ID’d game http://x.x.x.x:8080/?id=RANDOMID, it will fail as there isn’t a game with that ID (first part of the validation). This will also happen if we fail to start up the game.

The controller client can now listen for this ‘controller_connected’ event and give a message depending on its success:

io.on('controller_connected', function(connected) {

  if (connected) {

    alert("Connected!");

  } else {

    alert("Not connected!");
  }

});

Disconnecting

Now, the check to see if a game exists will work even if the game tab is closed before connecting the controller, since we haven’t put in socket disconnection events. So, let’s do that by adding this to the server code:

socket.on('disconnect', function () {

  // Game
  if (game_sockets[socket.id]) {

    console.log("Game disconnected");

    if (controller_sockets[game_sockets[socket.id].controller_id]) {
 
      controller_sockets[game_sockets[socket.id].controller_id].socket.emit("controller_connected", false);
      controller_sockets[game_sockets[socket.id].controller_id].game_id = undefined;
    }

    delete game_sockets[socket.id];
  }

  // Controller
  if (controller_sockets[socket.id]) {

    console.log("Controller disconnected");

    if (game_sockets[controller_sockets[socket.id].game_id]) {

      game_sockets[controller_sockets[socket.id].game_id].socket.emit("controller_connected", false);
      game_sockets[controller_sockets[socket.id].game_id].controller_id = undefined;
    }

    delete controller_sockets[socket.id];
  }
});

This checks whether the ID of the disconnected socket exists in the game or controller collection. It then uses the connected ID property (“game_id” if the socket is a controller, “controller_id” if the socket is a game) to notify the relevant socket of the disconnection, and removes it from the relevant socket reference. The socket disconnecting is then deleted. This means controllers cannot connect to games that have been closed down.

Now when a tab that is connected as a controller to a game is closed down, this should be in the terminal:

Controller disconnected

Adding a QR Code

If you have been manually typing the controller URL into your phone, you will be glad to know that it’s time to put in a QR code generator. We will be using this QR code generator.

Include this in the <head>:

<script src="//davidshimjs.github.com/qrcodejs/qrcode.min.js"></script>

Now in the else of where we check if the URL contains a parameter (where we emit game_connect) add this:

var qr = document.createElement('div');

qr.id = "qr";

document.body.appendChild(qr);

This creates an element with an ID of “qr” and appends it to the body.
Now where we are currently writing the URL into the body, replace document.body.innerHTML += url; with:

var qr_code = new QRCode("qr");
qr_code.makeCode(url);

This creates a QR Code (using the library) inside using our newly made div with the ID of qr from the provided URL.

Now refresh! Cool, huh?

The QR code is still present, even when the controller has connected. So, let’s fix that by adding this in the else (where we are doing the game code):

io.on('controller_connected', function(connected){

  if (connected) {

    qr.style.display = "none";

  }else{

    qr.style.display = "block";

  }

});

This alters the CSS of the QR code element when the controller_connected event is received. Now refresh! The QR code should now hide and show depending on the controller’s connectivity. Try disconnecting the controller.

Note: Your phone will have to be on the same internet connection as your computer. If you are seeing a 504 error on your phone, try adjusting your firewall settings.

Building the Car and the Floor

Good news. The hard part is all done! Now let’s have some fun with 3D.

First, the server must be able to serve up static files as we will be loading in a car model. In server.js add this anywhere in the “global” scope:

app.use("/public", express.static(__dirname + '/public'));

Create a public folder in the root and put the car.js (download it here) into it.

In index.html, include the 3D library in the head:

<script src="//threejs.org/build/three.min.js"></script>

Now to get the three scene set up. After where we declare the game_connected function, we have a bunch of config to add:

var renderer = new THREE.WebGLRenderer({
  antialias: true
}),
scene = new THREE.Scene(),
camera = new THREE.PerspectiveCamera(60, window.innerWidth / window.innerHeight, 1, 10000),

// Lights
ambient_light 		= new THREE.AmbientLight(0x222222),
directional_light = new THREE.DirectionalLight(0xffffff, 1),

// Used to load JSON models
loader = new THREE.JSONLoader(),

// Floor mesh
floor = new THREE.Mesh(new THREE.PlaneBufferGeometry(300,300), new THREE.MeshLambertMaterial({color: 0x22FF11})),

// Render loop
render = function(){

  // Render using scene and camera
  renderer.render(scene, camera);

  if (car)
    car.rotation.y += 0.01;

  // Call self
  requestAnimationFrame(render);
},
car;

// Enable shadows
renderer.shadowMapEnabled = true;

// Moves the camera "backward" (z) and "up" (y)
camera.position.z = -300;
camera.position.y = 100;

// Points the camera at the center of the floor
camera.lookAt(floor.position);

// Moves the directional light
directional_light.position.y = 150; // "up" / "down"
directional_light.position.x = -100; // "left" / "right"
directional_light.position.z = 60; // "forward" / "backward"

// Make the light able to cast shadows
directional_light.castShadow = true;

// Rotates the floor 90 degrees, so that it is horizontal
floor.rotation.x = -90 * (Math.PI / 180)

// Make the floor able to recieve shadows
floor.receiveShadow = true;

// Add camera, lights and floor to the scene
scene.add(camera);
scene.add(ambient_light);
scene.add(directional_light);
scene.add(floor);

// Load the car model
loader.load(
  'public/car.js',

  function ( geometry, materials ) {

    // Create the mesh from loaded geometry and materials
    var material = new THREE.MeshFaceMaterial( materials );
    car = new THREE.Mesh( geometry, material );

    // Can cast shadows
    car.castShadow = true;

    // Add to the scene
    scene.add( car );
  }
)

// Set size of renderer using window dimensions
renderer.setSize(window.innerWidth, window.innerHeight);

// Append to DOM
document.body.appendChild(renderer.domElement);

// This sets off the render loop
render();

This sets up the 3D scene, with a THREE plane mesh as a floor, two lights (one ambient and one directional, to cast shadows) and a loaded car model which rotates every requestAnimationFrame.

In detail, this declares the THREE components needed, positions the camera “up” and “back” a bit and rotates the camera using the .lookAt method which accepts a Vector3. The directional light is then positioned and instructed to cast shadows. This will make the light interact with meshes that have their castShadow or receiveShadow property set to true.

In this case we want the directional light and the car to cast shadows, and the floor to receive shadows.

The floor is rotated -90 degrees to make it “horizontal” to the camera and is set to receive shadows.

The camera, the lights and the floor are added to the scene. The renderer is set to the dimensions of the window and put into the DOM.

The car model is then requested, using the JSONLoader that comes with THREE, from the public directory. The callback function (which is fired once the model file is loaded) returns the geometry and the materials of the model, which is then used to create a mesh. It is set to cast shadows and is added to the scene.

Finally the render() loop is fired which renders (using the render method on the renderer) the scene using the camera, rotates the car if it has loaded (this is so that we know that the render loop is working correctly) and calls itself on requestAnimationFrame.

body {
  margin: 0;
}
#QR_code {
  position: absolute;
  top: 0;
  padding: 20px;
  background: white;
}

This removes the unwanted margin the canvas has by default and positions the QR code appropriately over the canvas element.

Here’s what we should see:

Controlling the Car

Now that the 3D scene is ready and the controller connection is working, it’s time to tie the two together. First, let’s create the events for the controller instance. In ‘controller_connected’, after the alert, add this:

var controller_state = {
  accelerate: false,
  steer: 0
},
emit_updates = function(){
  io.emit('controller_state_change', controller_state);
}
touchstart = function(e){
  e.preventDefault();

  controller_state.accelerate = true;
  emit_updates();
},
touchend = function(e){
  e.preventDefault();

  controller_state.accelerate = false;
  emit_updates();
},
devicemotion = function(e){
  controller_state.steer = e.accelerationIncludingGravity.y / 100;

  emit_updates();
}

document.body.addEventListener('touchstart', touchstart, false); // iOS & Android
document.body.addEventListener('MSPointerDown', touchstart, false); // Windows Phone
document.body.addEventListener('touchend', touchend, false); // iOS & Android
document.body.addEventListener('MSPointerUp', touchend, false); // Windows Phone
window.addEventListener('devicemotion', devicemotion, false);

This creates a controller_state object, and attaches events to the document body and the window. The accelerate property toggles between true and false on touchstart and touchend (the Windows Phone equivalent events are MSPointerDown and MSPointerUp). The steer property stores the tilt value of the phone on devicemotion.

In each of these functions, a custom event (controller_state_change) is emitted containing the current state of the controller.

Now that the controller client is sending its state when changed, the server needs to pass this information onto the relevant game. Add this into `server.js`, where a controller has successfully connected, so after:

game_sockets[game_socket_id].socket.emit("controller_connected", true);

Add this:

// Forward the changes onto the relative game socket
socket.on('controller_state_change', function(data) {

  if (game_sockets[game_socket_id]) {

    // Notify relevant game socket of controller state change
    game_sockets[game_socket_id].socket.emit("controller_state_change", data)
  }

});

Now that the server is forwarding the controller data on to the relevant game socket, it’s time to get the game client to listen for this and use the data. First we need a controller_state variable on the scope of the game instance so that it is accessible in the render (this will act as our gameloop). After where we declare a car placeholder, add this:

var speed = 0,
controller_state = {};

After the controller_connected listener in the game scope, add this:

// When the server sends a changed controller state update it in the game
io.on('controller_state_change', function(state) {

  controller_state = state;

});

This is the listener for when the server sends a new controller state, it updates the game’s controller state when the server sends a new one.

The game now has the controller state changing when the controller connected touches the screen and tilts the phone, but we aren’t using this data yet, replace:

if (car)
  car.rotation.y += 0.01;

with:

if (car) {

  // Rotate car
  if (controller_state.steer) {

    // Gives a number ranging from 0 to 1
    var percentage_speed = (speed / 2);

    // Rotate the car using the steer value
    // Multiplying it by the percentage speed makes the car not turn
    // unless accelerating and turns quicker as the speed increases.
    car.rotateY(controller_state.steer * percentage_speed);
  }

  // If controller is accelerating
  if (controller_state.accelerate) {

    // Add to speed until it is 2
    if (speed < 2) {
      speed += 0.05;
    } else {
      speed = 2;
    }

  // If controller is not accelerating
  } else {

    // Subtract from speed until 0
    if (0 < speed) {
      speed -= 0.05;
    } else {
      speed = 0;
    }
  }

  // Move car "forward" at speed
  car.translateZ(speed);

  // Collisions
  if (car.position.x > 150) {
    car.position.x = 150;
  }
  if (car.position.x < -150) {
    car.position.x = -150;
  }
  if (car.position.z > 150) {
    car.position.z = 150;
  }
  if (car.position.z < -150) {
    car.position.z = -150;
  }
}

This rotates the car using the (if present) steer property from the controller state, it is multiplied by a speed percentage (current speed/max speed), so that the car doesn’t rotate when stationary and the car rotates gradually quicker as the car’s speed increases.
The car moves forward using speed, which gradually increases or decreases depending on the accelerate property on the controller state.

The last part is for collisions, to stop the car going “off the floor”, for the sake of this demo it is hard coded so that the car’s x position stays between -150 and 150, the same goes for z.

Now the final thing to do is to reset the controller state when the controller has disconnected, after where we display the QR code again:

QR_code_element.style.display = "block";

add this:

controller_state = {};

Now the car should stop and reset its steering when the controller has disconnected.

Wrapping Up

Congrats on sticking with us this far! If the steering seems inverted, try rotating your phone 180 degrees, your steering wheel is upside down!

If you create something cool based on this, definitely let me know. Hit me up on Twitter at @cjonasw.

Evan Wieland

# April 22, 2015

This is really awesome!! Thanks Charlie!

Enis

Upon clicking the link to the live demo I get an “Application error”.

Dan

Permalink to comment# April 22, 2015

Me too. My guess is that they’ve hit their bandwidth allowance since the link was published.

Great example code and write-up, regardless.
Charlie Walter

Permalink to comment# April 22, 2015

Hey! Sorry about that, I’m looking into it! It seems to work after a few minutes though.

Verejan Daniel

That’s awesome!! Going to try this at home

Teelah

# April 23, 2015

Thanks for the share, will have to try this.

Mark

# April 24, 2015

Well… there goes my weekend! Amazing tutorial. Thanks for sharing.

Bogdan

# April 25, 2015

Awesome example! Thanks!

Josh

Thanks for the guide. I’m excited to give it a try!

# April 26, 2015

There seems to be a problem with Android smartphones when using the tutorial code, iOS and other browser instance events work. I can’t find the problem and I’ve taken a look at your live example where Android works fine.
Can you point me in the right direction?

Charlie Walter

Permalink to comment# April 28, 2015

Hi Bogdan,

Have you got the code up somewhere so I can take a look? Email me if you’d rather: [email protected]

Catalin

# April 28, 2015

This tutorial is very nice. Still you need to work on your article writing flow. It’s very unclear which code goes where. Maybe a download result link would’ve helped. I haven’t managed to make it work.

Charlie Walter

Permalink to comment# April 28, 2015

Hey,

I put the code up on Github https://github.com/cjonasw/smartphone-controlled-3d-web-game

I hope this helps

Charlie

Cierra Luke

Thanks for sharing this post. I am trying this. Up till now I did not get any issue. This web game tutorial is very helpful for a beginner like me.

Ranadip

Awesome tutorial, really loved the concept. Trying it out right away!

Santosh Shelke

# May 1, 2015

wow again good article. its realy helpfull. Thank yoy guys…