A Cooking Guidance Function on Android Tabletfor Homemade Cooking Assistance System

Yukiko Matsushima∗ Nobuo Funabiki∗ Tomoya Okada∗ Toru Nakanishi ∗ Kan Watanabe∗∗Department of Electrical and Communication Engineering, Okayama University

3-1-1 Tsushimanaka, Okayama 700-8530, Japan

Email: {funabiki,nakanisi,can}@cne.okayama-u.ac.jp, matsushima@sec.cne.okayama-u.ac.jp

Abstract—To assist busy people such as working persons,students, and nurturing families to have economical and healthylives with homemade cooking, we have proposed a Web-basedHOmemade Cooking Assistance System (HOCAS). Previously, weimplemented the menu planning function and the cooking-stepscheduling function at the server using JSP/Servlet, so that a usercan plan a menu for the whole week and make a cooking-stepschedule to efficiently cook multiple dishes at the same time.Unfortunately, a user needs to refer the next cooking-step ona paper during cooking process, which is actually not easy atcritical timing in a kitchen, because the schedule is output as aPDF file. In this paper, we propose a cooking guidance function onan Android tablet as a Java application to navigate cooking-stepsthrough touch panel operations. We confirmed the effectivenessby experimentally cooking four dishes using this function.

I. INTRODUCTION

The homemade cooking after working for long hours in

weekdays has become big burdens for busy people such as

working persons, students, and nurturing parents. Because the

homemade cooking usually needs several processes such as

a menu selection, a purchase of ingredients for the dishes,

and their cooking, it becomes very hard to take enough

time to complete them for dinners in weekdays. As a result,

busy people may rely on eating in restaurants or buying

processed/frozen foods, although they costs more. Besides,

they may not be able to achieve desired combinations of foods

that have been recommended in the food balance guide [1],

although the basic law on ”Shokuiku” has been enacted for

acquisition of knowledge about food and nutrition as well as

the ability to make appropriate food choices through various

experience related to food in order to develop people in the

ability to practice a healthy diet [2]. Then, they may become

a cause of the metabolic syndrome [3]. Thus, more and more

people have expected to have healthy and cost-efficient diets

by homemade cooking if the time is allowed.

To solve this time-shortage problem in a homemade cooking

in weekdays, we have proposed the two-phase cooking [4]. In

this two-phase cooking, first on a weekend, a person selects

the menus for the following weekdays, buys the necessary

ingredients of the dishes for the menus, and completes the

preparation cooking phases of the dishes. Then, in each

following weekday, he/she buys the additional ingredients that

are required for the dishes cooked in that day but could

not be bought beforehand in the weekend, and complete the

final cooking phases of the dishes in the menu. Thus, by

dividing the cooking procedures into those for a weekend

and for weekdays, we expect that the two-phase cooking can

drastically reduce the cooking time in weekdays.

To assist the two-phase cooking, we have proposed a menuplanning algorithm to select optimal one-week menus consid-

ering the constraints of cooking time and nutrition [5], and

a cooking-step scheduling algorithm to generate an optimal

schedule of applying the cooking-steps to make the dishes in

the menu [6][7]. In addition, we have proposed a Web-based

HOmemade Cooking Assistant System (HOCAS) to assist users

in using these algorithms from Web browsers [8].

Unfortunately, HOCAS does not have a function to navigate

the user in cooking the dishes in a kitchen by the generated

cooking schedule. Currently, the user needs to print out the

schedule and find the next cooking-step during the cooking.

Actually, it is not easy to refer the schedule on a paper and

follow it during cooking, because one schedule is composed

of a long list of various cooking steps to cook multiple dishes.

Particularly, it becomes almost impossible for a beginner

during cooking that is a target user of HOCAS. Besides, the

user has to measure the required time for each cooking step

manually and note it in a paper, because it is necessary to

make the schedule more accurate for the user.

In this paper, we propose a cooking guidance function to

navigate the user during cooking dishes in a kitchen and to ac-

curately record the starting/ending time of each cooking step.

We implement it as a Java application on an Android tablet,

instead of a Web application, because the Internet access is still

not common in a conventional kitchen, and the existing related

algorithms were implemented by Java. The cooking guidance

function allows paperless cooking navigations and automatic

recording of cooking time through touch panel operations.

The rest of this paper is organized as follows: Section II

overviews HOCAS. Section III proposes the cooking guidance

function. Section IV shows evaluation results of our proposal.

Section V discusses some related works. Section VI concludes

this paper with some future works.

II. OVERVIEW OF HOCAS

In previous studies, we have developed a Web application

called HOCAS (HOmemade Cooking Assistance System) to

assist busy people in cooking dishes by themselves at their

homes in weekdays. This section overviews the outline of

HOCAS.

IEEE R10-HTC2013 Sendai, Japan, August 26-29, 2013

249

A. Software Platform

HOCAS adopts Linux for the server OS, Tomcat for the

application server, and MySQL for the database. The HOCAS

programs are coded using Servlet and JSP. The user can

access to the HOCAS server through a Web browser. Figure 1

illustrates the platform of HOCAS.

Browser

HTML JSP Servlet

Tomcat

Linux

MySQL

Fig. 1. Software platform of HOCAS.

B. Existing Service Functions of HOCAS

Currently, HOCAS implements the menu planning functionand the cooking-step scheduling function for user services.

1) Menu Planning Function: The menu planning functionassists a user to generate a set of menus in the following

multiple days by using the menu planning algorithm that has

been proposed by our group [5]. The dates to be planed for in

one menu planning must be specified by the user using GoogleCalendar.

Our menu planning algorithm can generate a set of menus

that satisfy the four constraints: 1) the total cooking time in

any day must not exceed the limit specified by the user, 2) the

total volume of calories and salts must not exceed the user

limit, 3) any menu for one day must consists of main dishes

and side dishes, and 4) the type of any dish in one menu for a

day must be different from each other. We note that the type

represents the category of a dish such as a meat, a fish, and a

vegetable. Before applying this algorithm, the user is requested

to input the preference degree to each dish that he/she wants

to eat in the following days. Then, the algorithm selects dishes

that maximize the summation of the preference degrees under

the four constraints.

Figure 2 shows the interface for the user to select the

preference degrees for the dishes that he/she wants to eat.

For the dishes that the user does not want to eat, he/she does

not need to select. Then, HOCAS executes the algorithm to

generate menus, and outputs them into a CSV file and the

database. In addition, the user can see the menu of each day

using Google Calendar as shown in Figure 3. This allows the

user to access the menu from a mobile terminal so that he/she

can know the necessary ingredients for the menu at any time.

In addition to the abovementioned automatic generation

using the algorithm, this function allows the user to plan menus

manually. In this case, the user needs to select the dishes for

each day sequentially and to specify the dishes cooked in the

preparation cooking phase.

Main or SubDish name

Preference

Dish type Calorie & Salt

Fig. 2. Dish preference selection interface.

Location

Menu description

Google calendar

Fig. 3. Menu interface using Google Calendar.

2) Cooking-step Scheduling Function: The cooking-stepscheduling function generates an optimal schedule of cooking

the dishes for each day efficiently by applying the cooking-step scheduling algorithm that has been proposed by our group

[6][7]. In this algorithm, the recipe of any dish is regarded as a

sequence of cooking-steps that are classified into six different

ones, namely, Cut step, Mix step, Fry step, Boil step, Nuke step,

and Stand step. Before applying the algorithm, the number of

available items for each cooking utensil and the number of

cooks with the role of each cook should be specified by the

user. When two cooks participate in cooking, we assume that

one is a main cook who can handle any cooking-step, and

another is a sub cook who can handle only limited cooking-

steps.

Then, by considering the availability of a cook and/or the

necessary cooking utensil for the next cooking-step, a cooking-

step transition of a dish is determined. This cooking simulation

in the algorithm actually produces a schedule describing whoshould execute which cooking-step of a dish at when for a

given order of cooking dishes. Using the simulated anneal-

ing (SA), the dish cooking order is optimized by randomly

swapping the orders of two adjacent dishes.

Using the cooking-step scheduling function, the user can

generate both the one-day schedule and the multiple-dayschedule. The former schedule contains all the cooking-steps

of any dish in a menu, and does not consider the two-phase

IEEE R10-HTC2013 Sendai, Japan, August 26-29, 2013

250

cooking. The latter one generates the schedule for the cooking-

steps in preparation phases on a weekend and the schedules

for the cooking-steps in completion phases on the following

weekdays. This function outputs the result on a PDF file and

a CSV file. Figure 4 shows a sample output of this function

on a PDF file. The user needs to print the file on a paper and

refer it to cook the dishes.

0 min / main cook / Boiled pork

5 min / main cook / Pork miso soup

5 min / range0 / Boiled pork

10 min / main cook / Pork miso soup

Cooking schedule

Fig. 4. Cooking-step schedule on PDF file.

C. Utilization Flow of HOCAS

Figure 5 shows the utilization procedure of HOCAS. First,

the user has to register the name, login password, age, sex,

and exercise volume. The last three data is used to calculate

the limits on the calorie and the salt in the menu planning

algorithm. Besides, the user has to register the number of

cooking utensils such as a pot, a range, and a microwave that

are used in the cooking-step scheduling algorithm. Then, the

user can generate a menu and a cooking-step schedule by using

HOCAS.

III. PROPOSAL OF COOKING GUIDANCE FUNCTION

In this section, we propose a cooking guidance functionusing an Android tablet in HOCAS to navigate the cooking. In

this paper, we implemented this function as a Java application

on Android 4.0.4.

A. Overview

The function uses the outputs from the menu planning func-tion and the cooking-step scheduling function. In our current

implementation, the user needs to import them by copying the

corresponding CSV files to this function manually. We will

HOCAS

����cooking

user

manager

Fig. 5. Utilization flow of HOCAS.

implement the download function using a Web browser in our

future works. The cooking guidance function consists of the

menu display service, the cooking guidance service, and the

cooking-time measurement service. The following subsections

will discuss their details.

B. Menu Display Service

The menu display service shows a list of the dishes in a

menu and the list of the ingredients for each dish, so that

the user can confirm the details of the dishes that he/she will

cook. Figures 6 and 7 show examples of the dish list and the

ingredient list for a dish respectively.

Dish list

Tomato risotto with clam

Garlic & Miso batter of Chinese yam & hashed pork

Boiled & steamed Chinese cabbage & deep-fried tofu

Japanese style minestrone

Fig. 6. Dish list display.

C. Cooking Guidance Service

The cooking guidance service navigates the cooking process

of each cook (the main cook or the sub cook) by showing the

sequence of the cooking-steps given from the cooking-step

scheduling function. This function can switch the display for

both cooks, for the main cook only, and for the sub cook only

by clicking the corresponding button in the interface. Figure 8

shows this cook selection interface.

IEEE R10-HTC2013 Sendai, Japan, August 26-29, 2013

251

Tomato risotto with clam

Ingredients with volume

Fig. 7. Ingredient list display.

Elapsed time (total, main cook, sub cook)

Cooking guidance operation buttons

Fig. 8. Cook selection interface.

Figures 9, 10, and 11 show the interface of displaying the

sequence of the cooking-steps for the both cooks, for the main

cook, and for the sub cook respectively. In each interface, the

current cooking-step for the main cook is highlighted by red,

and that for the sub cook is by blue to avoid misrecognitions.

Besides, the color of the completed cooking-steps is changed

to gray from white. The start button and the end button in the

interface are prepared so that the user can signal the transition

of the cooking-step to the function. When either button is

clicked, the function changes the color of the corresponding

cooking-step.

D. Cooking-time Measurement Service

Using the interface in Figure 8, this function shows the

elapsed time for the whole cooking process and for each

cooking-step by a cook. The count of the elapsed time for

the whole cooking process is started when any start button in

the interfaces is first clicked. The count of the elapsed time

for each cooking-step is started when the corresponding start

button is clicked, and is ended when the end button is clicked.

The elapsed times for them are saved in the database to

improve the accuracy of the cooking-step scheduling algorithmby reflecting the real cooking-time for the user.

IV. EVALUATION BY COOKING EXPERIMENT

In this section, we evaluate the effectiveness of the cookingguidance function through an experiment of cooking four

Main cook at 0 minTomato risotto with clam

Main cook at 20 minJapanese style minestrone

Sub cook at 47 minTomato risotto with clam

Fig. 9. Cooking guidance interface for both cooks.

at 0 minTomato risotto with clam

at 20 minJapanese style minestrone

at 27 minJapanese style minestrone

Fig. 10. Cooking guidance interface for main cook.

dishes while using this function.

A. Experiment Setting

In this experiment, the four dishes in Table I were cooked

for four persons using an Android tablet with the cooking

guidance function. Two cooks participated in this experiment,

where the main cook can handle any cooking step, and the

sub cook can do only Mix step and Fry step among the

cooking steps that require operations of a cook. Three pots,

one microwave, and two ranges were used.

Figure 12 illustrates the cooking schedule generated by the

algorithm. This schedule indicates that the total cooking time

for all of the four dishes is 77 min.

B. Cooking Time Result

Table II shows the total cooking time estimated by the

algorithm in HOCAS and the one measured in this experiment.

IEEE R10-HTC2013 Sendai, Japan, August 26-29, 2013

252

TABLE IFOUR DISHES AND THEIR COOKING-STEPS IN EXPERIMENT.

dish order 1 2 3 4 5 6 7

Tomato risotto step Fry Fry Fry Boil Boil Mixwith clam time 7 10 10 15 10 5

Garlic and Miso batter step Cut Mix Fry Fry Fry Mix Mixof Chinese yam and hashed pork time 5 5 8 8 3 3 3

Boiled and steamed step Cut Cut Boil BoilChinese cabbage and deep-fried tofu time 10 5 3 2

Japanese style step Cut Cut Cut Boil Mixminestrone time 7 5 5 20 2

at 40 minTomato risotto with clam

at 47 minTomato risotto with clam

at 57 minTomato risotto with clam

Fig. 11. Cooking guidance interface for sub cook.

Although they are similar, there is some difference. Then, we

compared the time for six Cut steps in Table I. In any Cut step,

the estimated time is longer than the measured one, which may

cause this difference.

TABLE IITOTAL COOKING TIME (MIN).

algorithm experiment

77 82

TABLE IIICOOKING TIME FOR Cut steps (MIN.).

algorithm 7 5 5 10 5 5

experiment 4 3 3 4 3 3

Here, we discuss the effect of the time difference for Cutsteps in the whole cooking process. Figure 13 illustrates the

actual cooking schedule in this experiment. The comparison

with the one by the algorithm shows that actually, the used

range is different except for ”Japanese style minestrone”.

Because Cut steps finished earlier than the estimated, the main

cook used Range 2 for ”Boiled and steamed Chinese cabbage

and deep-fried tofu” earlier that for ”Garlic and Miso batter of

Chinese yam and hashed pork”, and continued to use it until

completion. As a result, no dishes can be cooked in parallel

at the latter half for using Range 1, and the total cooking time

became longer than that in the original schedule.

cut cutcut range1 mix

cut cut range1

� � �� � �� � �� � �2

� � �� � �� � �� � �2

� � �� � �� � �� � �2

� � �� � �� � �� � �2

� � �� � �� � �� � �2range2

cut mix� � � � � � � � � � � � � � � � � � � � � � � �

range1

77min.

stand

Fig. 12. Cooking schedule by algorithm in HOCAS.

cutcut range1 mix

cut

� � �� � �� � �� � �2

� � �� � �� � �� � �2

� � �� � �� � �� � �2

� � �� � �� � �� � �2

� � �� � �� � �� � �2range1

cut� � �� � �� � �� � �

2� � �� � �� � �� � �

2range2

cut

cut range2

82 min.����

����

����

����

����

mix

stand

Fig. 13. Cooking schedule in real cooking.

C. Reconstruction of Cooking Schedule

When the current situation in the real cooking is different

from the estimated one including the different cooking time

and the different range use, it can be effective to reconstruct the

cooking schedule dynamically. Thus, we actually reconstructed

the schedule by applying the cooking scheduling algorithm to

the remaining cooking-steps after Cut step 1 in Figure 13.

IEEE R10-HTC2013 Sendai, Japan, August 26-29, 2013

253

Figure 14 shows the reconstructed schedule that is identical to

the original one in Figure 12. This means that even if available,

the use of the different range from the scheduled one is not

correct and makes the cooking time longer.

cutcut range1cut

77 min.

mix

cut cut range1

� � �� � �� � �� � �2

� � �� � �� � �� � �2

� � �� � �� � �� � �2

� � �� � �� � �� � �2

cut mix� � � � � � � � � � � � � � � � � � � � � � � �

range2

range1

stand

����

Fig. 14. Reconstructed cooking schedule.

D. Future Works

The result in this experiment suggests that a cook can

easily make mistakes in selecting next cooking-steps and

instruments, and the cooking schedule in a real cooking can

become different from the scheduled one in HOCAS. As a

result, the total cooking time can become longer. To deal

with this problem, a voice guidance on the next step can

be effective, in addition to the current visual guidance in the

cooking guidance function on an Android tablet. Besides, a

reconstruction of the cooking schedule after recognizing the

current situation should be incorporated there.

Furthermore, Boil step and Stand step sometimes require

long time to keep the same step, whereas it is difficult for a

cook to follow the time accurately in a real cooking. Thus, a

time-up alarming should be added to inform a user the timing

of finishing the current step.

V. RELATED WORKS

In [9], Hamada et al. presented a method to create a process

flow graph automatically from textbooks for cooking programs

by creating a domain specific dictionary by statistical methods

and applying structural analysis methods using the dictionary.

In [10], Hamada et al. presented a cooking assistant software

called HappyCooking. The cooking process in a recipe is

divided into a sequence of primitive operations such as ”break

an egg” and ”bake an egg” for a process flow graph, and

generates an optimal schedule to minimize the total cooking

time using this graph. Unfortunately, it does not consider the

difference in cooking skills among multiple cooks.

In [11], Freyne et al. presented a preliminary study into the

suitability of recommender algorithms for recipe recommen-

dation based on preferences provided by 512 users on a corpus

of recipes. They examined the accuracy of collaborative,

and content-based filtering algorithms, and compared them to

hybrid recommender strategies.

In [12], Ueda et al. presented a method for extracting the

user’s preferences from his/her recipe browsing and cooking

history for a personalized recipe recommendation method

based on the food preferences in [13].

In [14] presented a tabletop dish recommendation system

for multiple participants dining together named the Group FDT

(Future Dining Table), which always recognizes the dining sta-

tus of the users by image processing, and recommends dishes

timely based on the investigation of real dining, literature, and

the experimental result.

VI. CONCLUSION

In this paper, we proposed a cooking guidance functionon an Android tablet in homemade cooking assistant system(HOCAS) to assist cooking process precisely. Through experi-

mentally cooking four dishes using this function, we confirmed

the effectiveness and some problems of it. In future works,

we will additionally implement a dynamic reconstruction of a

cooking schedule, a voice guidance, and a time-up alarming

for this function.

REFERENCES

[1] Ministry of Health, Labour and Welfare, Ministry of Health, Labour andWelfare, ”Food balance guide”, http://www.j-balanceguide.com/.

[2] Shokuiku, http://www.maff.go.jp/e/pdf/shokuiku.pdf.[3] Ministry of Health, Labour and Welfare, ”Preventive metabolic syn-

drome”, http://www.mhlw.go.jp/bunya/kenkou/metabo02/.[4] S. L. Ao, O. Castillo, and X. Huang ed., ”Intelligent control and innova-

tive computing (Chapter 10),” Springer, 2012. http://www.springer.com/engineering/control/book/978-1-4614-1694-4.

[5] N. Funabiki, S. Taniguchi, Y. Matsushima, and T. Nakanishi, ”Aproposal of a menu planning algorithm for two-phase cooking by busypersons,” Proc. Int. Work. Virtual Environment and Network-OrientedApplications (VENOA 2011), pp. 668-673, June 2011.

[6] Y. Matsushima, N. Funabiki, T. Nakanishi, and K. Watanabe, ”An appli-cation of cooking step scheduling algorithm for homemade cooking andits extensions,” Proc. Global Conf. Consumer Electronics (GCCE2012),pp. 94-98, October 2012.

[7] Y. Matsushima, N. Funabiki, T. Nakanishi, and K. Watanabe, ”Adynamic rescheduling extension of cooking-step scheduling algorithmfor multiple dish cooking,” Proc. Int. Conf. Applied and TheoreticalInformation Systems Research (2nd ATISR2012), CD-ROM, Dec. 2012.

[8] T. Okada, Y. Matsushima, S. Taniguchi, N. Funabiki, T. Nakanishi, andN. Amano, ”An implementation of menu planning and cooking stepoptimization functions in a Web-based homemade cooking assistancesystem,” IEICE Tech. Report, ET2010-128, pp. 205-210, March 2011.

[9] R. Hamada, I. Ide,, S. Sakai, and H. Tanaka, ”Structural analysis ofcooking preparation steps in Japanese,” Proc. Int. Workshop. Inform.Retrieval with Asian Lang., pp.157-164, 2000.

[10] R. Hamada, I. Ide, S. Sakai, and S. Sakai, ”HappyCooking: multimediacooking navigation software,” Digital Content Symposium (Japanese),June 2006.

[11] J. Freyne and S. Berkovsky, ”Recommending food: reasoning on recipesand ingredients,” Proc. Int. Conf. User Modeling, Adapt. Personal.(UMAP 2010), pp.381-386, 2010.

[12] M. Ueda, M. Takahata, and S. Nakajima, ”User’s food preference ex-traction for personalized cooking recipe recommendation,” Proc. Work.Semantic Personal. Inform. Manage.: Retrieval and Recommendation(SPIM 2011), 2011.

[13] M. Ueda, M. Takahata, and S. Nakajima, ”Recipe recommendationmethod based on user’s food preferences,” Proc. IADIS Int. Conf. e-Society, pp. 591-594, 2011.

[14] Y. Otsuka, J. Hu, and T. Inoue, ”Tabletop dish recommendation systemfor social dining: group FDT design based on the investigations of dishrecommendation,” J. Information Processing, vol. 21, no. 1, 2013.

IEEE R10-HTC2013 Sendai, Japan, August 26-29, 2013

254